Hive Query Failed

wrote:
*Hi Team ,*

We are using CM4.5.0 [ CDH4.2 + Hive 11 ]
I am getting below error message when i run big jobs using hive . I have
increased the Java Heap size but its not helping me to fix this issue .
Please do needful .


*stderr logs*

Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
at org.apache.hadoop.mapred.Child$3.run(Child.java:158)
Exception in thread "ResponseProcessor for block BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403" java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.addEntry(HashMap.java:753)
at java.util.HashMap.put(HashMap.java:385)
at java.util.HashSet.add(HashSet.java:200)
at sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:104)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:70)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:670)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

------------------------------


*syslog logs*

FO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100 rows
2014-03-08 00:56:41,151 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 5 forwarding 1000 rows
2014-03-08 00:56:41,152 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows
2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 5 forwarding 10000 rows
2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows
2014-03-08 00:56:57,036 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 1000 rows for join key [SIA285, WSSS-NZAA, 2013-08-09 22:47:00]
2014-03-08 00:59:45,131 INFO org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedReader.<init>(BufferedReader.java:80)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:515)
at org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:189)
at org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:812)
at org.apache.hadoop.mapred.Task.updateCounters(Task.java:834)
at org.apache.hadoop.mapred.Task.access$500(Task.java:72)
at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:661)
at java.lang.Thread.run(Thread.java:662)

2014-03-08 01:02:18,727 INFO org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: Java heap space

2014-03-08 01:02:18,728 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user.name (auth:SIMPLE) cause:org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
2014-03-08 01:02:26,632 WARN org.apache.hadoop.mapred.Child: Error running child
org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: EEXIST: File exists
at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
... 7 more
2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:518)
2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403 in pipeline xx.xx.xx.xx:50010, xx.xx.xx.xx:50010: bad datanode xx.xx.xx.xx:50010
2014-03-08 01:06:19,453 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task



Regards,

Jagan M

To unsubscribe from this group and stop receiving emails from it, send
an email to scm-users+...@cloudera.org <javascript:>.

On Saturday, March 8, 2014 10:01:51 AM UTC+5:30, Darren Lo wrote:

Hi Jagan,

It looks like the memory of the MapReduce job is insufficient. Which java

heap size did you increase? I'd check your MapReduce Gateway
configurations
for the various heaps and make sure those are configured to have
sufficient
heap. Be sure to deploy client configurations to make sure your
/etc/hadoop/conf picks up your changes.

cdh-...@cloudera.org <javascript:> may be able to offer better tips.

Thanks,
Darren


On Fri, Mar 7, 2014 at 6:15 PM, Jagan Madhappan
<madha...@gmail.com<javascript:>

On 3/8/14, Jagan Madhappan wrote:
Darren ,
Thanks for your response .
My mappers completed successfully but i am facing issue in reduce phase .
I have increased heap size using CM & changed below property
Gateway --> Resource Management ->MapReduce Child Java Maximum Heap Size to
1GB then restarted job tracker & ran deploy client configuration

Regards,
Jagan M

<property>
<name>mapred.child.java.opts</name>
<value> -Xmx1073741824</value>
</property>

In which property i have to change for increase reducer heap size
using CM4.5.0 because my mappers were completed without any issue only
reducers creating an issue

Regards,
Jagan
java
have
.
java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403"
sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:104)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:70)
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:670)
rows
org.apache.hadoop.hive.ql.exec.JoinOperator:
org.apache.hadoop.hive.ql.exec.JoinOperator:
rows
Communication
org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:515)
org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:189)
org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:812)
Communication
EEXIST:
File
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62)
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153)
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114)
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:518)
BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403
send
To unsubscribe from this group and stop receiving emails from it, send an
email to scm-users+unsubscribe@cloudera.org.