FAQ
Hi Jim,

I checked the source codes. There is no way to get exceptions in these two line in DFSClient. Are you using candidate 0?
1506: private long bytesCurBlock = 0; // bytes writen in current block
2262: if (currentPacket != null) {

Nicholas

----- Original Message ----
From: Jim Kellerman <jim@powerset.com>
To: "core-dev@hadoop.apache.org" <core-dev@hadoop.apache.org>
Sent: Sunday, March 30, 2008 2:58:26 PM
Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

-1

HBase-0.1.0 has a torture test we run to qualify releases. This test ran fine on Hadoop-0.16.[01]
but failed on Hadoop-0.16.2 release candidate with the following exceptions:

2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020] hbase.HLog(380): Could not append to log. Opening new log. Exception:
java.io.IOException: Stream closed.
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.(DFSClient.java:382)
at org.apache.hadoop.dfs.DistributedFileSystem.create(DistributedFileSystem.java:123)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:436)
at org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:379)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
at org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionServer.java:539)
2008-03-30 17:32:58,993 ERROR [regionserver/0:0:0:0:0:0:0:0:8020.logRoller] hbase.HRegionServer$LogRoller(545): Log rolling failed
java.lang.NullPointerException
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2262)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:51)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:67)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
at org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionServer.java:539)

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Friday, March 28, 2008 10:19 PM
To: core-dev@hadoop.apache.org
Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

I've created a candidate build for Hadoop 0.16.2. This fixes
25 issues in 0.16.1. Now that HBase has it's own releases,
it has been removed from this release.

Please download and try it out.

http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

Should we release this? Vote closes on Wednesday, April 2.

Nigel



No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
Date: 3/28/2008 10:58 AM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM

Search Discussions

  • Jim Kellerman at Mar 31, 2008 at 5:21 am
    Yes, I am using candidate 0 which I downloaded from http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: s29752-hadoopdev@yahoo.com
    Sent: Sunday, March 30, 2008 6:30 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Hi Jim,

    I checked the source codes. There is no way to get
    exceptions in these two line in DFSClient. Are you using candidate 0?
    1506: private long bytesCurBlock = 0; // bytes writen in
    current block
    2262: if (currentPacket != null) {

    Nicholas

    ----- Original Message ----
    From: Jim Kellerman <jim@powerset.com>
    To: "core-dev@hadoop.apache.org" <core-dev@hadoop.apache.org>
    Sent: Sunday, March 30, 2008 2:58:26 PM
    Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    -1

    HBase-0.1.0 has a torture test we run to qualify releases.
    This test ran fine on Hadoop-0.16.[01] but failed on
    Hadoop-0.16.2 release candidate with the following exceptions:

    2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
    hbase.HLog(380): Could not append to log. Opening new log. Exception:
    java.io.IOException: Stream closed.
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
    nt.java:1506)
    at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
    at
    org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
    FileSystem.java:123)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:436)
    at
    org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:827)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)
    2008-03-30 17:32:58,993 ERROR
    [regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
    hbase.HRegionServer$LogRoller(545): Log rolling failed
    java.lang.NullPointerException
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
    t.java:2262)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
    DataOutputStream.java:51)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
    am.java:67)
    at
    org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Friday, March 28, 2008 10:19 PM
    To: core-dev@hadoop.apache.org
    Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    I've created a candidate build for Hadoop 0.16.2. This fixes
    25 issues in 0.16.1. Now that HBase has it's own releases, it has
    been removed from this release.

    Please download and try it out.

    http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    Should we release this? Vote closes on Wednesday, April 2.

    Nigel



    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
    Date: 3/28/2008 10:58 AM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM





    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM
  • Nigel Daley at Mar 31, 2008 at 5:31 am
    Jim, did you recompile hbase with this new release? The fact that
    the line numbers don't match means something is probably running
    using an old version.

    Nige
    On Mar 30, 2008, at 10:21 PM, Jim Kellerman wrote:

    Yes, I am using candidate 0 which I downloaded from http://
    people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: s29752-hadoopdev@yahoo.com
    Sent: Sunday, March 30, 2008 6:30 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Hi Jim,

    I checked the source codes. There is no way to get
    exceptions in these two line in DFSClient. Are you using
    candidate 0?
    1506: private long bytesCurBlock = 0; // bytes writen in
    current block
    2262: if (currentPacket != null) {

    Nicholas

    ----- Original Message ----
    From: Jim Kellerman <jim@powerset.com>
    To: "core-dev@hadoop.apache.org" <core-dev@hadoop.apache.org>
    Sent: Sunday, March 30, 2008 2:58:26 PM
    Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    -1

    HBase-0.1.0 has a torture test we run to qualify releases.
    This test ran fine on Hadoop-0.16.[01] but failed on
    Hadoop-0.16.2 release candidate with the following exceptions:

    2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
    hbase.HLog(380): Could not append to log. Opening new log. Exception:
    java.io.IOException: Stream closed.
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
    nt.java:1506)
    at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
    at
    org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
    FileSystem.java:123)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
    436)
    at
    org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:
    827)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)
    2008-03-30 17:32:58,993 ERROR
    [regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
    hbase.HRegionServer$LogRoller(545): Log rolling failed
    java.lang.NullPointerException
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
    t.java:2262)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
    DataOutputStream.java:51)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
    am.java:67)
    at
    org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Friday, March 28, 2008 10:19 PM
    To: core-dev@hadoop.apache.org
    Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    I've created a candidate build for Hadoop 0.16.2. This fixes
    25 issues in 0.16.1. Now that HBase has it's own releases, it has
    been removed from this release.

    Please download and try it out.

    http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    Should we release this? Vote closes on Wednesday, April 2.

    Nigel



    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
    Date: 3/28/2008 10:58 AM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM





    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
    3/30/2008 12:32 PM
  • Jim Kellerman at Mar 31, 2008 at 6:13 am
    I did not recompile hbase-0.1.0 and it ran fine with hadoop-0.16.[01]. I made two runs against 0.16.2 and they both failed the same way.

    However, I just re-ran against 0.16.2 and the test passed, so I must have configured something incorrectly.
    My bad!.

    +1 for 0.16.2

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Sunday, March 30, 2008 10:31 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Jim, did you recompile hbase with this new release? The fact
    that the line numbers don't match means something is probably
    running using an old version.

    Nige
    On Mar 30, 2008, at 10:21 PM, Jim Kellerman wrote:

    Yes, I am using candidate 0 which I downloaded from http://
    people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: s29752-hadoopdev@yahoo.com
    Sent: Sunday, March 30, 2008 6:30 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Hi Jim,

    I checked the source codes. There is no way to get exceptions in
    these two line in DFSClient. Are you using candidate 0?
    1506: private long bytesCurBlock = 0; // bytes writen in
    current block
    2262: if (currentPacket != null) {

    Nicholas

    ----- Original Message ----
    From: Jim Kellerman <jim@powerset.com>
    To: "core-dev@hadoop.apache.org" <core-dev@hadoop.apache.org>
    Sent: Sunday, March 30, 2008 2:58:26 PM
    Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    -1

    HBase-0.1.0 has a torture test we run to qualify releases.
    This test ran fine on Hadoop-0.16.[01] but failed on
    Hadoop-0.16.2 release candidate with the following exceptions:

    2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
    hbase.HLog(380): Could not append to log. Opening new log.
    Exception:
    java.io.IOException: Stream closed.
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
    nt.java:1506)
    at
    org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
    at
    org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
    FileSystem.java:123)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
    436)
    at
    org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:
    827)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)
    2008-03-30 17:32:58,993 ERROR
    [regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
    hbase.HRegionServer$LogRoller(545): Log rolling failed
    java.lang.NullPointerException
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
    t.java:2262)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
    DataOutputStream.java:51)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
    am.java:67)
    at
    org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Friday, March 28, 2008 10:19 PM
    To: core-dev@hadoop.apache.org
    Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    I've created a candidate build for Hadoop 0.16.2. This fixes
    25 issues in 0.16.1. Now that HBase has it's own
    releases, it has
    been removed from this release.

    Please download and try it out.

    http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    Should we release this? Vote closes on Wednesday, April 2.

    Nigel



    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
    Date: 3/28/2008 10:58 AM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM





    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
    3/30/2008 12:32 PM

    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM
  • Nigel Daley at Mar 31, 2008 at 6:21 am
    Perhaps it's an intermittent problem? Can you run a couple more times?
    On Mar 30, 2008, at 11:12 PM, Jim Kellerman wrote:

    I did not recompile hbase-0.1.0 and it ran fine with hadoop-0.16.
    [01]. I made two runs against 0.16.2 and they both failed the same
    way.

    However, I just re-ran against 0.16.2 and the test passed, so I
    must have configured something incorrectly.
    My bad!.

    +1 for 0.16.2

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Sunday, March 30, 2008 10:31 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Jim, did you recompile hbase with this new release? The fact
    that the line numbers don't match means something is probably
    running using an old version.

    Nige
    On Mar 30, 2008, at 10:21 PM, Jim Kellerman wrote:

    Yes, I am using candidate 0 which I downloaded from http://
    people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: s29752-hadoopdev@yahoo.com
    Sent: Sunday, March 30, 2008 6:30 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Hi Jim,

    I checked the source codes. There is no way to get exceptions in
    these two line in DFSClient. Are you using candidate 0?
    1506: private long bytesCurBlock = 0; // bytes writen in
    current block
    2262: if (currentPacket != null) {

    Nicholas

    ----- Original Message ----
    From: Jim Kellerman <jim@powerset.com>
    To: "core-dev@hadoop.apache.org" <core-dev@hadoop.apache.org>
    Sent: Sunday, March 30, 2008 2:58:26 PM
    Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    -1

    HBase-0.1.0 has a torture test we run to qualify releases.
    This test ran fine on Hadoop-0.16.[01] but failed on
    Hadoop-0.16.2 release candidate with the following exceptions:

    2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
    hbase.HLog(380): Could not append to log. Opening new log.
    Exception:
    java.io.IOException: Stream closed.
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
    nt.java:1506)
    at
    org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
    at
    org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
    FileSystem.java:123)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
    436)
    at
    org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:
    827)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)
    2008-03-30 17:32:58,993 ERROR
    [regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
    hbase.HRegionServer$LogRoller(545): Log rolling failed
    java.lang.NullPointerException
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
    t.java:2262)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
    DataOutputStream.java:51)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
    am.java:67)
    at
    org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Friday, March 28, 2008 10:19 PM
    To: core-dev@hadoop.apache.org
    Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    I've created a candidate build for Hadoop 0.16.2. This fixes
    25 issues in 0.16.1. Now that HBase has it's own
    releases, it has
    been removed from this release.

    Please download and try it out.

    http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    Should we release this? Vote closes on Wednesday, April 2.

    Nigel



    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
    Date: 3/28/2008 10:58 AM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM





    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
    3/30/2008 12:32 PM

    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
    3/30/2008 12:32 PM
  • Jim Kellerman at Mar 31, 2008 at 6:58 am
    Yes, it does appear to be intermittent. hbase-0.1.0 was built against hadoop-0.16.0, so I assumed that there were no incompatible changes. It just failed again with the exception:

    2008-03-31 06:42:29,079 ERROR [regionserver/0:0:0:0:0:0:0:0:8020.compactor] hbase.HRegionServer$CompactSplitThread(267): Compaction failed for region TestTable,0021017156,1206945717128
    java.lang.NullPointerException
    at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2262)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:51)
    at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:67)
    at org.apache.hadoop.hbase.HStoreFile.writeSplitInfo(HStoreFile.java:267)
    at org.apache.hadoop.hbase.HStoreFile.writeReferenceFiles(HStoreFile.java:246)
    at org.apache.hadoop.hbase.HStoreFile.splitStoreFile(HStoreFile.java:239)
    at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:740)
    at org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.split(HRegionServer.java:291)
    at org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.run(HRegionServer.java:255)

    and if you look at the source for DFSClient for 0.16.0, line 2262 is:
    2260: synchronized (dataQueue) {
    2261: if (response != null) {
    2262: response.close();
    2263: }

    Obviously the code in DFSClient$DFSOutputStream has changed considerably from 0.16.0 to 0.16.2.

    My assumption was that applications built against 0.16.x should not have to be recompiled for 0.16.y. Is that not correct?

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Sunday, March 30, 2008 11:20 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Perhaps it's an intermittent problem? Can you run a couple
    more times?
    On Mar 30, 2008, at 11:12 PM, Jim Kellerman wrote:

    I did not recompile hbase-0.1.0 and it ran fine with hadoop-0.16.
    [01]. I made two runs against 0.16.2 and they both failed the same
    way.

    However, I just re-ran against 0.16.2 and the test passed, so I must
    have configured something incorrectly.
    My bad!.

    +1 for 0.16.2

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Sunday, March 30, 2008 10:31 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Jim, did you recompile hbase with this new release? The fact that
    the line numbers don't match means something is probably running
    using an old version.

    Nige
    On Mar 30, 2008, at 10:21 PM, Jim Kellerman wrote:

    Yes, I am using candidate 0 which I downloaded from http://
    people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: s29752-hadoopdev@yahoo.com
    Sent: Sunday, March 30, 2008 6:30 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Hi Jim,

    I checked the source codes. There is no way to get
    exceptions in
    these two line in DFSClient. Are you using candidate 0?
    1506: private long bytesCurBlock = 0; // bytes writen in
    current block
    2262: if (currentPacket != null) {

    Nicholas

    ----- Original Message ----
    From: Jim Kellerman <jim@powerset.com>
    To: "core-dev@hadoop.apache.org" <core-dev@hadoop.apache.org>
    Sent: Sunday, March 30, 2008 2:58:26 PM
    Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    -1

    HBase-0.1.0 has a torture test we run to qualify releases.
    This test ran fine on Hadoop-0.16.[01] but failed on
    Hadoop-0.16.2 release candidate with the following exceptions:

    2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
    hbase.HLog(380): Could not append to log. Opening new log.
    Exception:
    java.io.IOException: Stream closed.
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
    nt.java:1506)
    at
    org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
    at
    org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
    FileSystem.java:123)
    at
    org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
    436)
    at
    org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:
    827)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)
    2008-03-30 17:32:58,993 ERROR
    [regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
    hbase.HRegionServer$LogRoller(545): Log rolling failed
    java.lang.NullPointerException
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
    t.java:2262)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
    DataOutputStream.java:51)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
    am.java:67)
    at
    org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
    at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Friday, March 28, 2008 10:19 PM
    To: core-dev@hadoop.apache.org
    Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    I've created a candidate build for Hadoop 0.16.2. This fixes
    25 issues in 0.16.1. Now that HBase has it's own
    releases, it has
    been removed from this release.

    Please download and try it out.

    http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    Should we release this? Vote closes on Wednesday, April 2.

    Nigel



    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
    Date: 3/28/2008 10:58 AM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM





    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
    3/30/2008 12:32 PM

    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
    3/30/2008 12:32 PM

    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM
  • Jim Kellerman at Mar 31, 2008 at 9:07 am
    Assuming that applications built against 0.16.x should not have to be recompiled for 0.16.x+1, I have to assume that an intermittent bug has been introduced between hadoop 0.16.1 and 0.16.2 because I have run the HBase torture test 6 times against both 0.16.1 and 0.16.2. Hadoop 0.16.2 failed 3 out of 6 times.
    Hadoop 0.16.1 succeeded 6 out of 6 times.

    If a recompile between 0.16.1 and 0.16.2 is required, then 0.16.2 gets a -1.

    If no recompile should be required, then 0.16.2 gets a -1 because it fails
    50% of the time where 0.16.1 succeeded.

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Jim Kellerman
    Sent: Sunday, March 30, 2008 11:58 PM
    To: core-dev@hadoop.apache.org
    Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Yes, it does appear to be intermittent. hbase-0.1.0 was built
    against hadoop-0.16.0, so I assumed that there were no
    incompatible changes. It just failed again with the exception:

    2008-03-31 06:42:29,079 ERROR
    [regionserver/0:0:0:0:0:0:0:0:8020.compactor]
    hbase.HRegionServer$CompactSplitThread(267): Compaction
    failed for region TestTable,0021017156,1206945717128
    java.lang.NullPointerException
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
    t.java:2262)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
    DataOutputStream.java:51)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
    am.java:67)
    at
    org.apache.hadoop.hbase.HStoreFile.writeSplitInfo(HStoreFile.java:267)
    at
    org.apache.hadoop.hbase.HStoreFile.writeReferenceFiles(HStoreF
    ile.java:246)
    at
    org.apache.hadoop.hbase.HStoreFile.splitStoreFile(HStoreFile.java:239)
    at
    org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:740)
    at
    org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.split
    (HRegionServer.java:291)
    at
    org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.run(H
    RegionServer.java:255)

    and if you look at the source for DFSClient for 0.16.0, line 2262 is:
    2260: synchronized (dataQueue) {
    2261: if (response != null) {
    2262: response.close();
    2263: }

    Obviously the code in DFSClient$DFSOutputStream has changed
    considerably from 0.16.0 to 0.16.2.

    My assumption was that applications built against 0.16.x
    should not have to be recompiled for 0.16.y. Is that not correct?

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Sunday, March 30, 2008 11:20 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Perhaps it's an intermittent problem? Can you run a couple more
    times?
    On Mar 30, 2008, at 11:12 PM, Jim Kellerman wrote:

    I did not recompile hbase-0.1.0 and it ran fine with hadoop-0.16.
    [01]. I made two runs against 0.16.2 and they both failed
    the same
    way.

    However, I just re-ran against 0.16.2 and the test passed, so I must
    have configured something incorrectly.
    My bad!.

    +1 for 0.16.2

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Sunday, March 30, 2008 10:31 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Jim, did you recompile hbase with this new release? The
    fact that
    the line numbers don't match means something is probably running
    using an old version.

    Nige
    On Mar 30, 2008, at 10:21 PM, Jim Kellerman wrote:

    Yes, I am using candidate 0 which I downloaded from http://
    people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: s29752-hadoopdev@yahoo.com
    Sent: Sunday, March 30, 2008 6:30 PM
    To: core-dev@hadoop.apache.org
    Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    Hi Jim,

    I checked the source codes. There is no way to get
    exceptions in
    these two line in DFSClient. Are you using candidate 0?
    1506: private long bytesCurBlock = 0; // bytes writen in
    current block
    2262: if (currentPacket != null) {

    Nicholas

    ----- Original Message ----
    From: Jim Kellerman <jim@powerset.com>
    To: "core-dev@hadoop.apache.org" <core-dev@hadoop.apache.org>
    Sent: Sunday, March 30, 2008 2:58:26 PM
    Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    -1

    HBase-0.1.0 has a torture test we run to qualify releases.
    This test ran fine on Hadoop-0.16.[01] but failed on
    Hadoop-0.16.2 release candidate with the following exceptions:

    2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
    hbase.HLog(380): Could not append to log. Opening new log.
    Exception:
    java.io.IOException: Stream closed.
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
    nt.java:1506)
    at
    org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
    at
    org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
    FileSystem.java:123)
    at
    org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
    436)
    at
    org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:
    827)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
    at
    org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
    at
    org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)
    2008-03-30 17:32:58,993 ERROR
    [regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
    hbase.HRegionServer$LogRoller(545): Log rolling failed
    java.lang.NullPointerException
    at
    org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
    t.java:2262)
    at
    org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
    DataOutputStream.java:51)
    at
    org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
    am.java:67)
    at
    org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
    at
    org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
    at
    org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
    ver.java:539)

    ---
    Jim Kellerman, Senior Engineer; Powerset

    -----Original Message-----
    From: Nigel Daley
    Sent: Friday, March 28, 2008 10:19 PM
    To: core-dev@hadoop.apache.org
    Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

    I've created a candidate build for Hadoop 0.16.2. This fixes
    25 issues in 0.16.1. Now that HBase has it's own
    releases, it has
    been removed from this release.

    Please download and try it out.

    http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

    Should we release this? Vote closes on Wednesday, April 2.

    Nigel



    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
    Date: 3/28/2008 10:58 AM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM





    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
    3/30/2008 12:32 PM

    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
    3/30/2008 12:32 PM

    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM


    No virus found in this incoming message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
    Date: 3/30/2008 12:32 PM
    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM
  • Jim Kellerman at Mar 31, 2008 at 4:11 pm
    It appears that the classpath was incorrect during the test runs where hadoop-0.16.2 had failures.

    With the classpath corrected, hadoop-0.16.2 passed the HBase stress test 5 times out of 5.

    +1 for hadoop-0.16.2

    -------------
    About the HBase stress test:

    The HBase stress test puts both Hadoop and HBase under heavy load, which is why we use it to qualify HBase releases.

    In this test we run a 4 node cluster. All 4 nodes are data nodes.
    Node xxx-xxx-12.u.powerset.com runs the name node, JobTracker and
    HBase master. The test is also launched from this node. All 4 nodes
    run task trackers.

    The sole region server runs on xxx-xxx-13.u.powerset.com

    The application (PerformanceEvaluation) is launched from xxx-xxx-12.u.powerset.com but runs map jobs on all 4 nodes of the cluster.

    We install everything on -12 and use rsync to make sure all the cluster
    nodes are doing the same thing.

    Attachments:
    env.sh sets up the directories we want to rsync/run from.
    rsync.sh is how we push the software around the cluster.
    start.sh is how we start up hadoop and hbase.
    The command line for running PerformanceEvaluation is:

    $ date | tee start.time ; hbase-0.1.0/bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4 ; date | tee end.time

    hadoop-env.sh - how we point hadoop at hbase
    hadoop-site.xml - just changes port numbers and hadoop.tmp.dir
    masters - points to xxx-xxx-12.u.powerset.com
    slaves - points to xxx-xxx-1[2-5].u.powerset.com

    For HBase:
    hbase-site.xml changes port numbers
    regionservers - points to xxx-xxx-13.u.powerset.com
    hbase-env.sh - points to JAVA_HOME and HBASE-HOME (only modifications from distribution)

    The key seems to be in $HBASE_HOME/bin/hbase
    - This file includes all jars in $HBASE_HOME/lib - because that directory contains hadoop jars, they get put on the classpath which would explain the strange behavior and mis-matched line numbers. hbase-0.1.0 ships with jars from hadoop-0.16.0. Replacing the 0.16.0 jars in HBASE_HOME/lib with the jars from hadoop-0.16.2 seems to have solved the problem.

    My bad.
    ---
    Jim Kellerman, Senior Engineer; Powerset

    No virus found in this outgoing message.
    Checked by AVG.
    Version: 7.5.519 / Virus Database: 269.22.1/1352 - Release Date: 3/31/2008 10:13 AM

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-dev @
categorieshadoop
postedMar 31, '08 at 1:30a
activeMar 31, '08 at 4:11p
posts8
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase