[Hadoop-common-dev] Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

[Hadoop-common-dev] Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

	S29752-hadoopdev
	Mar 31, 2008 at 1:30 am

Hi Jim,

I checked the source codes. There is no way to get exceptions in these two line in DFSClient. Are you using candidate 0?
1506: private long bytesCurBlock = 0; // bytes writen in current block
2262: if (currentPacket != null) {

Nicholas

----- Original Message ----
From: Jim Kellerman <[email protected]>
To: "cor[email protected]" <[email protected]>
Sent: Sunday, March 30, 2008 2:58:26 PM
Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

-1

HBase-0.1.0 has a torture test we run to qualify releases. This test ran fine on Hadoop-0.16.[01]
but failed on Hadoop-0.16.2 release candidate with the following exceptions:

2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020] hbase.HLog(380): Could not append to log. Opening new log. Exception:
java.io.IOException: Stream closed.
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.(DFSClient.java:382)
at org.apache.hadoop.dfs.DistributedFileSystem.create(DistributedFileSystem.java:123)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:436)
at org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:379)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
at org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionServer.java:539)
2008-03-30 17:32:58,993 ERROR [regionserver/0:0:0:0:0:0:0:0:8020.logRoller] hbase.HRegionServer$LogRoller(545): Log rolling failed
java.lang.NullPointerException
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2262)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:51)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:67)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
at org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionServer.java:539)

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Friday, March 28, 2008 10:19 PM
To: [email protected]
Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

I've created a candidate build for Hadoop 0.16.2. This fixes
25 issues in 0.16.1. Now that HBase has it's own releases,
it has been removed from this release.

Please download and try it out.

http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

Should we release this? Vote closes on Wednesday, April 2.

Nigel

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
Date: 3/28/2008 10:58 AM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM

Search Discussions

7 responses
Oldest
Nested

Jim Kellerman

at Mar 31, 2008 at 5:21 am

⇧

Yes, I am using candidate 0 which I downloaded from http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: [email protected]
Sent: Sunday, March 30, 2008 6:30 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Hi Jim,

I checked the source codes. There is no way to get
exceptions in these two line in DFSClient. Are you using candidate 0?
1506: private long bytesCurBlock = 0; // bytes writen in
current block
2262: if (currentPacket != null) {

Nicholas

----- Original Message ----
From: Jim Kellerman <[email protected]>
To: "cor[email protected]" <[email protected]>
Sent: Sunday, March 30, 2008 2:58:26 PM
Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

-1

HBase-0.1.0 has a torture test we run to qualify releases.
This test ran fine on Hadoop-0.16.[01] but failed on
Hadoop-0.16.2 release candidate with the following exceptions:

2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
hbase.HLog(380): Could not append to log. Opening new log. Exception:
java.io.IOException: Stream closed.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
nt.java:1506)
at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
at
org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:436)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:827)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)
2008-03-30 17:32:58,993 ERROR
[regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
hbase.HRegionServer$LogRoller(545): Log rolling failed
java.lang.NullPointerException
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
t.java:2262)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
DataOutputStream.java:51)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
am.java:67)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Friday, March 28, 2008 10:19 PM
To: [email protected]
Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

I've created a candidate build for Hadoop 0.16.2. This fixes
25 issues in 0.16.1. Now that HBase has it's own releases, it has
been removed from this release.

Please download and try it out.

http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

Should we release this? Vote closes on Wednesday, April 2.

Nigel

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
Date: 3/28/2008 10:58 AM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM

reply | permalink

Nigel Daley Jim, did you recompile hbase with this new release? The fact that the line numbers don't match means something is probably running using an old version. Nige

Nigel Daley at Mar 31, 2008 at 5:31 am ⇧

Jim, did you recompile hbase with this new release? The fact that
the line numbers don't match means something is probably running
using an old version.

Nige
On Mar 30, 2008, at 10:21 PM, Jim Kellerman wrote:

Yes, I am using candidate 0 which I downloaded from http://
people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: [email protected]
Sent: Sunday, March 30, 2008 6:30 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Hi Jim,

I checked the source codes. There is no way to get
exceptions in these two line in DFSClient. Are you using
candidate 0?
1506: private long bytesCurBlock = 0; // bytes writen in
current block
2262: if (currentPacket != null) {

Nicholas

----- Original Message ----
From: Jim Kellerman <[email protected]>
To: "cor[email protected]" <[email protected]>
Sent: Sunday, March 30, 2008 2:58:26 PM
Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

-1

HBase-0.1.0 has a torture test we run to qualify releases.
This test ran fine on Hadoop-0.16.[01] but failed on
Hadoop-0.16.2 release candidate with the following exceptions:

2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
hbase.HLog(380): Could not append to log. Opening new log. Exception:
java.io.IOException: Stream closed.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
nt.java:1506)
at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
at
org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
436)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:
827)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)
2008-03-30 17:32:58,993 ERROR
[regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
hbase.HRegionServer$LogRoller(545): Log rolling failed
java.lang.NullPointerException
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
t.java:2262)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
DataOutputStream.java:51)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
am.java:67)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Friday, March 28, 2008 10:19 PM
To: [email protected]
Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

I've created a candidate build for Hadoop 0.16.2. This fixes
25 issues in 0.16.1. Now that HBase has it's own releases, it has
been removed from this release.

Please download and try it out.

http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

Should we release this? Vote closes on Wednesday, April 2.

Nigel

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
Date: 3/28/2008 10:58 AM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
3/30/2008 12:32 PM

reply | permalink

Jim Kellerman

at Mar 31, 2008 at 6:13 am

⇧

I did not recompile hbase-0.1.0 and it ran fine with hadoop-0.16.[01]. I made two runs against 0.16.2 and they both failed the same way.

However, I just re-ran against 0.16.2 and the test passed, so I must have configured something incorrectly.
My bad!.

+1 for 0.16.2

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Sunday, March 30, 2008 10:31 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Jim, did you recompile hbase with this new release? The fact
that the line numbers don't match means something is probably
running using an old version.

Nige

On Mar 30, 2008, at 10:21 PM, Jim Kellerman wrote:

Yes, I am using candidate 0 which I downloaded from http://
people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: [email protected]

Sent: Sunday, March 30, 2008 6:30 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Hi Jim,

I checked the source codes. There is no way to get exceptions in
these two line in DFSClient. Are you using candidate 0?
1506: private long bytesCurBlock = 0; // bytes writen in
current block
2262: if (currentPacket != null) {

Nicholas

----- Original Message ----
From: Jim Kellerman <[email protected]>
To: "cor[email protected]" <[email protected]>
Sent: Sunday, March 30, 2008 2:58:26 PM
Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

-1

HBase-0.1.0 has a torture test we run to qualify releases.
This test ran fine on Hadoop-0.16.[01] but failed on
Hadoop-0.16.2 release candidate with the following exceptions:

2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
hbase.HLog(380): Could not append to log. Opening new log.

Exception:

java.io.IOException: Stream closed.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
nt.java:1506)
at

org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)

at
org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
436)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:
827)
at

org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)

org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)

at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)
2008-03-30 17:32:58,993 ERROR
[regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
hbase.HRegionServer$LogRoller(545): Log rolling failed
java.lang.NullPointerException
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
t.java:2262)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
DataOutputStream.java:51)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
am.java:67)
at

org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)

at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)

---
Jim Kellerman, Senior Engineer; Powerset

releases, it has

been removed from this release.

Please download and try it out.

http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

Should we release this? Vote closes on Wednesday, April 2.

Nigel

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
Date: 3/28/2008 10:58 AM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM

reply | permalink

Nigel Daley Perhaps it's an intermittent problem? Can you run a couple more times?

Nigel Daley at Mar 31, 2008 at 6:21 am ⇧

Perhaps it's an intermittent problem? Can you run a couple more times?
On Mar 30, 2008, at 11:12 PM, Jim Kellerman wrote:

I did not recompile hbase-0.1.0 and it ran fine with hadoop-0.16.
[01]. I made two runs against 0.16.2 and they both failed the same
way.

However, I just re-ran against 0.16.2 and the test passed, so I
must have configured something incorrectly.
My bad!.

+1 for 0.16.2

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Sunday, March 30, 2008 10:31 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Jim, did you recompile hbase with this new release? The fact
that the line numbers don't match means something is probably
running using an old version.

Nige
On Mar 30, 2008, at 10:21 PM, Jim Kellerman wrote:

Yes, I am using candidate 0 which I downloaded from http://
people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: [email protected]
Sent: Sunday, March 30, 2008 6:30 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Hi Jim,

I checked the source codes. There is no way to get exceptions in
these two line in DFSClient. Are you using candidate 0?
1506: private long bytesCurBlock = 0; // bytes writen in
current block
2262: if (currentPacket != null) {

Nicholas

----- Original Message ----
From: Jim Kellerman <[email protected]>
To: "cor[email protected]" <[email protected]>
Sent: Sunday, March 30, 2008 2:58:26 PM
Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

-1

HBase-0.1.0 has a torture test we run to qualify releases.
This test ran fine on Hadoop-0.16.[01] but failed on
Hadoop-0.16.2 release candidate with the following exceptions:

2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
hbase.HLog(380): Could not append to log. Opening new log.
Exception:
java.io.IOException: Stream closed.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
nt.java:1506)
at
org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)
at
org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:
436)
at
org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:
827)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)
at
org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)
2008-03-30 17:32:58,993 ERROR
[regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
hbase.HRegionServer$LogRoller(545): Log rolling failed
java.lang.NullPointerException
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
t.java:2262)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
DataOutputStream.java:51)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
am.java:67)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)
at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Friday, March 28, 2008 10:19 PM
To: [email protected]
Subject: [VOTE] Release Hadoop 0.16.2 (candidate 0)

I've created a candidate build for Hadoop 0.16.2. This fixes
25 issues in 0.16.1. Now that HBase has it's own
releases, it has
been removed from this release.

Please download and try it out.

http://people.apache.org/~nigel/hadoop-0.16.2-candidate-0/

Should we release this? Vote closes on Wednesday, April 2.

Nigel

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1348 - Release
Date: 3/28/2008 10:58 AM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
3/30/2008 12:32 PM

reply | permalink

Jim Kellerman

at Mar 31, 2008 at 6:58 am

⇧

Yes, it does appear to be intermittent. hbase-0.1.0 was built against hadoop-0.16.0, so I assumed that there were no incompatible changes. It just failed again with the exception:

2008-03-31 06:42:29,079 ERROR [regionserver/0:0:0:0:0:0:0:0:8020.compactor] hbase.HRegionServer$CompactSplitThread(267): Compaction failed for region TestTable,0021017156,1206945717128
java.lang.NullPointerException
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2262)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:51)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:67)
at org.apache.hadoop.hbase.HStoreFile.writeSplitInfo(HStoreFile.java:267)
at org.apache.hadoop.hbase.HStoreFile.writeReferenceFiles(HStoreFile.java:246)
at org.apache.hadoop.hbase.HStoreFile.splitStoreFile(HStoreFile.java:239)
at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:740)
at org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.split(HRegionServer.java:291)
at org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.run(HRegionServer.java:255)

and if you look at the source for DFSClient for 0.16.0, line 2262 is:
2260: synchronized (dataQueue) {
2261: if (response != null) {
2262: response.close();
2263: }

Obviously the code in DFSClient$DFSOutputStream has changed considerably from 0.16.0 to 0.16.2.

My assumption was that applications built against 0.16.x should not have to be recompiled for 0.16.y. Is that not correct?

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Sunday, March 30, 2008 11:20 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Perhaps it's an intermittent problem? Can you run a couple
more times?

On Mar 30, 2008, at 11:12 PM, Jim Kellerman wrote:

I did not recompile hbase-0.1.0 and it ran fine with hadoop-0.16.
[01]. I made two runs against 0.16.2 and they both failed the same
way.

However, I just re-ran against 0.16.2 and the test passed, so I must
have configured something incorrectly.
My bad!.

+1 for 0.16.2

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Nigel Daley
Sent: Sunday, March 30, 2008 10:31 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Jim, did you recompile hbase with this new release? The fact that
the line numbers don't match means something is probably running
using an old version.

Nige

-----Original Message-----
From: [email protected]

Sent: Sunday, March 30, 2008 6:30 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Hi Jim,

I checked the source codes. There is no way to get

exceptions in

these two line in DFSClient. Are you using candidate 0?
1506: private long bytesCurBlock = 0; // bytes writen in
current block
2262: if (currentPacket != null) {

Nicholas

----- Original Message ----
From: Jim Kellerman <[email protected]>
To: "cor[email protected]" <[email protected]>
Sent: Sunday, March 30, 2008 2:58:26 PM
Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

-1

HBase-0.1.0 has a torture test we run to qualify releases.
This test ran fine on Hadoop-0.16.[01] but failed on
Hadoop-0.16.2 release candidate with the following exceptions:

2008-03-30 17:32:58,993 ERROR [IPC Server handler 5 on 8020]
hbase.HLog(380): Could not append to log. Opening new log.

Exception:

java.io.IOException: Stream closed.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
nt.java:1506)
at

org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)

at
org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
FileSystem.java:123)
at

org.apache.hadoop.fs.FileSystem.create(FileSystem.java:

436)
at

org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:

827)
at

org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)

org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)

org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)

at org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)
at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)

---
Jim Kellerman, Senior Engineer; Powerset

releases, it has

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM

reply | permalink

Jim Kellerman

at Mar 31, 2008 at 9:07 am

⇧

Assuming that applications built against 0.16.x should not have to be recompiled for 0.16.x+1, I have to assume that an intermittent bug has been introduced between hadoop 0.16.1 and 0.16.2 because I have run the HBase torture test 6 times against both 0.16.1 and 0.16.2. Hadoop 0.16.2 failed 3 out of 6 times.
Hadoop 0.16.1 succeeded 6 out of 6 times.

If a recompile between 0.16.1 and 0.16.2 is required, then 0.16.2 gets a -1.

If no recompile should be required, then 0.16.2 gets a -1 because it fails
50% of the time where 0.16.1 succeeded.

---
Jim Kellerman, Senior Engineer; Powerset

-----Original Message-----
From: Jim Kellerman
Sent: Sunday, March 30, 2008 11:58 PM
To: [email protected]
Subject: RE: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Yes, it does appear to be intermittent. hbase-0.1.0 was built
against hadoop-0.16.0, so I assumed that there were no
incompatible changes. It just failed again with the exception:

2008-03-31 06:42:29,079 ERROR
[regionserver/0:0:0:0:0:0:0:0:8020.compactor]
hbase.HRegionServer$CompactSplitThread(267): Compaction
failed for region TestTable,0021017156,1206945717128
java.lang.NullPointerException
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
t.java:2262)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
DataOutputStream.java:51)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
am.java:67)
at
org.apache.hadoop.hbase.HStoreFile.writeSplitInfo(HStoreFile.java:267)
at
org.apache.hadoop.hbase.HStoreFile.writeReferenceFiles(HStoreF
ile.java:246)
at
org.apache.hadoop.hbase.HStoreFile.splitStoreFile(HStoreFile.java:239)
at
org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:740)
at
org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.split
(HRegionServer.java:291)
at
org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.run(H
RegionServer.java:255)

and if you look at the source for DFSClient for 0.16.0, line 2262 is:
2260: synchronized (dataQueue) {
2261: if (response != null) {
2262: response.close();
2263: }

Obviously the code in DFSClient$DFSOutputStream has changed
considerably from 0.16.0 to 0.16.2.

My assumption was that applications built against 0.16.x
should not have to be recompiled for 0.16.y. Is that not correct?

---
Jim Kellerman, Senior Engineer; Powerset

On Mar 30, 2008, at 11:12 PM, Jim Kellerman wrote:

I did not recompile hbase-0.1.0 and it ran fine with hadoop-0.16.
[01]. I made two runs against 0.16.2 and they both failed

the same

way.

However, I just re-ran against 0.16.2 and the test passed, so I must
have configured something incorrectly.
My bad!.

+1 for 0.16.2

---
Jim Kellerman, Senior Engineer; Powerset

fact that

the line numbers don't match means something is probably running
using an old version.

Nige

-----Original Message-----
From: [email protected]

Sent: Sunday, March 30, 2008 6:30 PM
To: [email protected]
Subject: Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Hi Jim,

I checked the source codes. There is no way to get

exceptions in

Exception:

java.io.IOException: Stream closed.
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(DFSClie
nt.java:1506)
at

org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:382)

at
org.apache.hadoop.dfs.DistributedFileSystem.create(Distributed
FileSystem.java:123)
at

org.apache.hadoop.fs.FileSystem.create(FileSystem.java:

436)
at

org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:

827)
at

org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:379)

org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270)

org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:230)

at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)
2008-03-30 17:32:58,993 ERROR
[regionserver/0:0:0:0:0:0:0:0:8020.logRoller]
hbase.HRegionServer$LogRoller(545): Log rolling failed
java.lang.NullPointerException
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClien
t.java:2262)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FS
DataOutputStream.java:51)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStre
am.java:67)
at

org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:932)

org.apache.hadoop.hbase.HLog.rollWriter(HLog.java:217)

at
org.apache.hadoop.hbase.HRegionServer$LogRoller.run(HRegionSer
ver.java:539)

---
Jim Kellerman, Senior Engineer; Powerset

releases, it has

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date:
3/30/2008 12:32 PM

No virus found in this incoming message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release
Date: 3/30/2008 12:32 PM

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1350 - Release Date: 3/30/2008 12:32 PM

reply | permalink

Jim Kellerman

at Mar 31, 2008 at 4:11 pm

⇧

It appears that the classpath was incorrect during the test runs where hadoop-0.16.2 had failures.

With the classpath corrected, hadoop-0.16.2 passed the HBase stress test 5 times out of 5.

+1 for hadoop-0.16.2

-------------
About the HBase stress test:

The HBase stress test puts both Hadoop and HBase under heavy load, which is why we use it to qualify HBase releases.

In this test we run a 4 node cluster. All 4 nodes are data nodes.
Node xxx-xxx-12.u.powerset.com runs the name node, JobTracker and
HBase master. The test is also launched from this node. All 4 nodes
run task trackers.

The sole region server runs on xxx-xxx-13.u.powerset.com

The application (PerformanceEvaluation) is launched from xxx-xxx-12.u.powerset.com but runs map jobs on all 4 nodes of the cluster.

We install everything on -12 and use rsync to make sure all the cluster
nodes are doing the same thing.

Attachments:
env.sh sets up the directories we want to rsync/run from.
rsync.sh is how we push the software around the cluster.
start.sh is how we start up hadoop and hbase.
The command line for running PerformanceEvaluation is:

$ date | tee start.time ; hbase-0.1.0/bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 4 ; date | tee end.time

hadoop-env.sh - how we point hadoop at hbase
hadoop-site.xml - just changes port numbers and hadoop.tmp.dir
masters - points to xxx-xxx-12.u.powerset.com
slaves - points to xxx-xxx-1[2-5].u.powerset.com

For HBase:
hbase-site.xml changes port numbers
regionservers - points to xxx-xxx-13.u.powerset.com
hbase-env.sh - points to JAVA_HOME and HBASE-HOME (only modifications from distribution)

The key seems to be in $HBASE_HOME/bin/hbase
- This file includes all jars in $HBASE_HOME/lib - because that directory contains hadoop jars, they get put on the classpath which would explain the strange behavior and mis-matched line numbers. hbase-0.1.0 ships with jars from hadoop-0.16.0. Replacing the 0.16.0 jars in HBASE_HOME/lib with the jars from hadoop-0.16.2 seems to have solved the problem.

My bad.
---
Jim Kellerman, Senior Engineer; Powerset

No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.519 / Virus Database: 269.22.1/1352 - Release Date: 3/31/2008 10:13 AM

reply | permalink

Related Discussions

Discussion Navigation

view	thread \| post

Discussion Overview

group	common-dev @
categories	hadoop
posted	Mar 31, '08 at 1:30a
active	Mar 31, '08 at 4:11p
posts	8
users	3
website	hadoop.apache.org...
irc	#hadoop

responses	expanded	Hotkey: s
font	variable	Hotkey: f
user style	avatars	Hotkey: a

[Hadoop-common-dev] Re: [VOTE] Release Hadoop 0.16.2 (candidate 0)

Search Discussions

Related Discussions

3 users in discussion