FAQ
I managed to resolve the issue by bumping up the value
on ipc.maximum.data.length. Apparently this would be caused by having a
block report too long and is due to too many small files.
On Friday, October 18, 2013 9:45:49 PM UTC-7, Jason Mason wrote:

Hey,

Just upgrading distro from 4.2.1 to 4.4. Restarted namenodes and
datanodes and now seeing the exceptions below. Namenode has been in safe
mode for an hour now. Not sure if these are something I should be worried
about?

On the namenode:

2013-10-18 23:41:20,138 WARN org.apache.hadoop.ipc.Server: Requested data
length 135758849 is longer than maximum configured RPC length 67108864.
RPC came from <ip>
2013-10-18 23:41:03,225 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 8020: readAndProcess threw exception java.io.IOException:
Requested data length 194413907 is longer than maximum configured RPC
length 67108864. RPC came from <ip> from client <ip>. Count of bytes read:

java.io.IOException: Requested data length 194413907 is longer than
maximum configured RPC length 67108864. RPC came from <ip>
at
org.apache.hadoop.ipc.Server$Connection.checkDataLength(Server.java:1341)
at
org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1439)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:758)
at
org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:557)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:532)

On the datanodes:

2013-10-18 23:42:48,785 INFO org.apache.hadoop.util.JvmPauseMonitor:
Detected pause in JVM or host machine (eg GC): pause of approximately 1382ms
GC pool 'ParNew' had collection(s): count=1 time=1398ms
2013-10-18 23:42:49,995 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
java.io.IOException: Failed on local exception: java.io.IOException:
Response is null.; Host Details : local host is: "<domain>/<ip>";
destination host is: "<domain>":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:763)
at org.apache.hadoop.ipc.Client.call(Client.java:1241)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy10.blockReport(Unknown Source)
at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy10.blockReport(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:201)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:399)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:551)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Response is null.
at
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:951)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:846)
--

---
You received this message because you are subscribed to the Google Groups "CDH Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cdh-user+unsubscribe@cloudera.org.
For more options, visit https://groups.google.com/a/cloudera.org/groups/opt_out.

Search Discussions

Discussion Posts

Previous

Follow ups

Related Discussions

Discussion Navigation
viewthread | post
posts ‹ prev | 2 of 4 | next ›
Discussion Overview
groupcdh-user @
categorieshadoop
postedOct 19, '13 at 4:45a
activeMay 14, '14 at 6:29p
posts4
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Jason Mason: 2 posts Gal Halevy: 2 posts

People

Translate

site design / logo © 2022 Grokbase