FAQ
I have a small 3 node Ubuntu 12.04 cluster setup with the latest version of
CM4. This is running in a virtual environment, so I'd like to add a new
node by clone one of the existing nodes. So far after cloning, I just
change the hostname then add it to the cluster. I assign roles, and most
services seem to start fine except hdfs. I assume that old data might be a
problem. Perhaps I should also delete the /dfs directory before joining the
cloned system to the cluster? Anything I should cleanup?

Currently I see warnings like:

"Failed to delete replica blk_-6472368801651805991_1563: ReplicaInfo not
found."

Then exceptions:

Error processing datanode Command
java.io.IOException: Error in deleting blocks.
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:1109)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:571)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:535)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:702)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:535)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:674)
at java.lang.Thread.run(Thread.java:662)


And finally:


Block pool BP-757795456-10.0.0.253-1349243581960 (storage id
DS-1820226960-10.0.0.1-50010-1349243591090) service to
manager.hadoop.local/10.0.0.253:8020 is shutting down
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnregisteredNodeException):
Data node DatanodeRegistration(10.0.0.3,
storageID=DS-1820226960-10.0.0.1-50010-1349243591090, infoPort=50075,
ipcPort=50020, storageInfo=lv=-40;cid=cluster8;nsid=1300054773;c=0) is
attempting to report storage ID DS-1820226960-10.0.0.1-50010-1349243591090.
Node 10.0.0.1:50010 is expected to serve this storage.
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanode(DatanodeManager.java:254)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processIncrementalBlockReport(BlockManager.java:2588)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReceivedAndDeleted(NameNodeRpcServer.java:895)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolServerSideTranslatorPB.java:181)
at
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:20024)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)

at org.apache.hadoop.ipc.Client.call(Client.java:1160)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy10.blockReceivedAndDeleted(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy10.blockReceivedAndDeleted(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:226)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reportReceivedDeletedBlocks(BPServiceActor.java:281)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:547)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:674)
at java.lang.Thread.run(Thread.java:662)


And the DataNode shuts down.

Search Discussions

Discussion Posts

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 1 of 2 | next ›
Discussion Overview
groupscm-users @
categorieshadoop
postedOct 12, '12 at 6:11a
activeOct 13, '12 at 11:01p
posts2
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Brad Schick: 1 post Joey Echeverria: 1 post

People

Translate

site design / logo © 2022 Grokbase