FAQ
*Hi Team ,*

We are using CM4.5.0 [ CDH4.2 + Hive 11 ]
I am getting below error message when i run big jobs using hive . I have
increased the Java Heap size but its not helping me to fix this issue .
Please do needful .


*stderr logs*

Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError: Java heap space
  at java.io.BufferedOutputStream.(BufferedOutputStream.java:42)
  at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
  at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
  at org.apache.hadoop.mapred.Child$3.run(Child.java:158)
Exception in thread "ResponseProcessor for block BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403" java.lang.OutOfMemoryError: Java heap space
  at java.util.HashMap.addEntry(HashMap.java:753)
  at java.util.HashMap.put(HashMap.java:385)
  at java.util.HashSet.add(HashSet.java:200)
  at sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:104)
  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:70)
  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
  at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
  at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
  at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
  at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
  at java.io.FilterInputStream.read(FilterInputStream.java:66)
  at java.io.FilterInputStream.read(FilterInputStream.java:66)
  at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
  at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:670)
log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

------------------------------


*syslog logs*

FO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100 rows
2014-03-08 00:56:41,151 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 5 forwarding 1000 rows
2014-03-08 00:56:41,152 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows
2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 5 forwarding 10000 rows
2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows
2014-03-08 00:56:57,036 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 1000 rows for join key [SIA285, WSSS-NZAA, 2013-08-09 22:47:00]
2014-03-08 00:59:45,131 INFO org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: Java heap space
  at java.io.BufferedReader.(BufferedReader.java:91)
  at org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:515)
  at org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:189)
  at org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
  at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:812)
  at org.apache.hadoop.mapred.Task.updateCounters(Task.java:834)
  at org.apache.hadoop.mapred.Task.access$500(Task.java:72)
  at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:661)
  at java.lang.Thread.run(Thread.java:662)

2014-03-08 01:02:18,727 INFO org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: Java heap space

2014-03-08 01:02:18,728 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user.name (auth:SIMPLE) cause:org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
2014-03-08 01:02:26,632 WARN org.apache.hadoop.mapred.Child: Error running child
org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
  at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
  at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
  at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
  at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:396)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
  at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: EEXIST: File exists
  at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
  at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
  ... 7 more
2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
java.io.IOException: Broken pipe
  at sun.nio.ch.FileDispatcher.write0(Native Method)
  at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
  at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
  at sun.nio.ch.IOUtil.write(IOUtil.java:40)
  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
  at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62)
  at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
  at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153)
  at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114)
  at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
  at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
  at java.io.DataOutputStream.flush(DataOutputStream.java:106)
  at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:518)
2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403 in pipeline xx.xx.xx.xx:50010, xx.xx.xx.xx:50010: bad datanode xx.xx.xx.xx:50010
2014-03-08 01:06:19,453 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task



Regards,

Jagan M

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Search Discussions

  • Darren Lo at Mar 8, 2014 at 4:31 am
    Hi Jagan,

    It looks like the memory of the MapReduce job is insufficient. Which java
    heap size did you increase? I'd check your MapReduce Gateway configurations
    for the various heaps and make sure those are configured to have sufficient
    heap. Be sure to deploy client configurations to make sure your
    /etc/hadoop/conf picks up your changes.

    cdh-user@cloudera.org may be able to offer better tips.

    Thanks,
    Darren

    On Fri, Mar 7, 2014 at 6:15 PM, Jagan Madhappan wrote:


    *Hi Team ,*

    We are using CM4.5.0 [ CDH4.2 + Hive 11 ]
    I am getting below error message when i run big jobs using hive . I have
    increased the Java Heap size but its not helping me to fix this issue .
    Please do needful .


    *stderr logs*

    Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
    at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:158)
    Exception in thread "ResponseProcessor for block BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403" java.lang.OutOfMemoryError: Java heap space
    at java.util.HashMap.addEntry(HashMap.java:753)
    at java.util.HashMap.put(HashMap.java:385)
    at java.util.HashSet.add(HashSet.java:200)
    at sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:104)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:70)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:670)
    log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

    ------------------------------


    *syslog logs*

    FO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100 rows
    2014-03-08 00:56:41,151 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 5 forwarding 1000 rows
    2014-03-08 00:56:41,152 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows
    2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 5 forwarding 10000 rows
    2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows
    2014-03-08 00:56:57,036 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 1000 rows for join key [SIA285, WSSS-NZAA, 2013-08-09 22:47:00]
    2014-03-08 00:59:45,131 INFO org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedReader.<init>(BufferedReader.java:80)
    at java.io.BufferedReader.<init>(BufferedReader.java:91)
    at org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:515)
    at org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:189)
    at org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
    at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:812)
    at org.apache.hadoop.mapred.Task.updateCounters(Task.java:834)
    at org.apache.hadoop.mapred.Task.access$500(Task.java:72)
    at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:661)
    at java.lang.Thread.run(Thread.java:662)

    2014-03-08 01:02:18,727 INFO org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: Java heap space

    2014-03-08 01:02:18,728 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user.name (auth:SIMPLE) cause:org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
    2014-03-08 01:02:26,632 WARN org.apache.hadoop.mapred.Child: Error running child
    org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
    at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
    at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: EEXIST: File exists
    at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
    at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
    ... 7 more
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
    java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
    at sun.nio.ch.IOUtil.write(IOUtil.java:40)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
    at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
    at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153)
    at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:518)
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403 in pipeline xx.xx.xx.xx:50010, xx.xx.xx.xx:50010: bad datanode xx.xx.xx.xx:50010
    2014-03-08 01:06:19,453 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task



    Regards,

    Jagan M

    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Jagan Madhappan at Mar 8, 2014 at 1:24 pm
    Darren ,
    Thanks for your response .
    My mappers completed successfully but i am facing issue in reduce phase .
    I have increased heap size using CM & changed below property
    Gateway --> Resource Management ->MapReduce Child Java Maximum Heap Size to
    1GB then restarted job tracker & ran deploy client configuration

    Regards,
    Jagan M

    On Saturday, March 8, 2014 10:01:51 AM UTC+5:30, Darren Lo wrote:

    Hi Jagan,

    It looks like the memory of the MapReduce job is insufficient. Which java
    heap size did you increase? I'd check your MapReduce Gateway configurations
    for the various heaps and make sure those are configured to have sufficient
    heap. Be sure to deploy client configurations to make sure your
    /etc/hadoop/conf picks up your changes.

    cdh-...@cloudera.org <javascript:> may be able to offer better tips.

    Thanks,
    Darren


    On Fri, Mar 7, 2014 at 6:15 PM, Jagan Madhappan <madha...@gmail.com<javascript:>
    wrote:
    *Hi Team ,*

    We are using CM4.5.0 [ CDH4.2 + Hive 11 ]
    I am getting below error message when i run big jobs using hive . I have
    increased the Java Heap size but its not helping me to fix this issue .
    Please do needful .


    *stderr logs*

    Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
    at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:158)
    Exception in thread "ResponseProcessor for block BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403" java.lang.OutOfMemoryError: Java heap space
    at java.util.HashMap.addEntry(HashMap.java:753)
    at java.util.HashMap.put(HashMap.java:385)
    at java.util.HashSet.add(HashSet.java:200)
    at sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:104)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:70)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
    at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:670)
    log4j:WARN No appenders could be found for logger (org.apache.hadoop.hdfs.DFSClient).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

    ------------------------------


    *syslog logs*

    FO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100 rows
    2014-03-08 00:56:41,151 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 5 forwarding 1000 rows
    2014-03-08 00:56:41,152 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows
    2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 5 forwarding 10000 rows
    2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows
    2014-03-08 00:56:57,036 INFO org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 1000 rows for join key [SIA285, WSSS-NZAA, 2013-08-09 22:47:00]
    2014-03-08 00:59:45,131 INFO org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedReader.<init>(BufferedReader.java:80)
    at java.io.BufferedReader.<init>(BufferedReader.java:91)
    at org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:515)
    at org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:189)
    at org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
    at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:812)
    at org.apache.hadoop.mapred.Task.updateCounters(Task.java:834)
    at org.apache.hadoop.mapred.Task.access$500(Task.java:72)
    at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:661)
    at java.lang.Thread.run(Thread.java:662)

    2014-03-08 01:02:18,727 INFO org.apache.hadoop.mapred.Task: Communication exception: java.lang.OutOfMemoryError: Java heap space

    2014-03-08 01:02:18,728 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user.name (auth:SIMPLE) cause:org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
    2014-03-08 01:02:26,632 WARN org.apache.hadoop.mapred.Child: Error running child
    org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File exists
    at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
    at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: EEXIST: File exists
    at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
    at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
    ... 7 more
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
    java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
    at sun.nio.ch.IOUtil.write(IOUtil.java:40)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
    at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
    at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153)
    at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:518)
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403 in pipeline xx.xx.xx.xx:50010, xx.xx.xx.xx:50010: bad datanode xx.xx.xx.xx:50010
    2014-03-08 01:06:19,453 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task



    Regards,

    Jagan M

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Jagan Madhappan at Mar 8, 2014 at 1:31 pm
    /etc/hadoop/conf file has been updated .

      <property>
         <name>mapred.child.java.opts</name>
         <value> -Xmx1073741824</value>
       </property>

    In which property i have to change for increase reducer heap size
    using CM4.5.0 because my mappers were completed without any issue only
    reducers creating an issue

    Regards,
    Jagan
    On 3/8/14, Jagan Madhappan wrote:
    Darren ,
    Thanks for your response .
    My mappers completed successfully but i am facing issue in reduce phase .
    I have increased heap size using CM & changed below property
    Gateway --> Resource Management ->MapReduce Child Java Maximum Heap Size to

    1GB then restarted job tracker & ran deploy client configuration

    Regards,
    Jagan M

    On Saturday, March 8, 2014 10:01:51 AM UTC+5:30, Darren Lo wrote:

    Hi Jagan,

    It looks like the memory of the MapReduce job is insufficient. Which java

    heap size did you increase? I'd check your MapReduce Gateway
    configurations
    for the various heaps and make sure those are configured to have
    sufficient
    heap. Be sure to deploy client configurations to make sure your
    /etc/hadoop/conf picks up your changes.

    cdh-...@cloudera.org <javascript:> may be able to offer better tips.

    Thanks,
    Darren


    On Fri, Mar 7, 2014 at 6:15 PM, Jagan Madhappan
    <madha...@gmail.com<javascript:>
    wrote:
    *Hi Team ,*

    We are using CM4.5.0 [ CDH4.2 + Hive 11 ]
    I am getting below error message when i run big jobs using hive . I have

    increased the Java Heap size but its not helping me to fix this issue .
    Please do needful .


    *stderr logs*

    Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError:
    Java heap space
    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
    at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
    at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:158)
    Exception in thread "ResponseProcessor for block
    BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403"
    java.lang.OutOfMemoryError: Java heap space
    at java.util.HashMap.addEntry(HashMap.java:753)
    at java.util.HashMap.put(HashMap.java:385)
    at java.util.HashSet.add(HashSet.java:200)
    at
    sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:104)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:70)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at
    org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at
    org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
    at
    org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
    at
    org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:670)
    log4j:WARN No appenders could be found for logger
    (org.apache.hadoop.hdfs.DFSClient).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
    more info.

    ------------------------------


    *syslog logs*

    FO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100 rows
    2014-03-08 00:56:41,151 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
    5 forwarding 1000 rows
    2014-03-08 00:56:41,152 INFO
    org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows
    2014-03-08 00:56:54,609 INFO org.apache.hadoop.hive.ql.exec.JoinOperator:
    5 forwarding 10000 rows
    2014-03-08 00:56:54,609 INFO
    org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows
    2014-03-08 00:56:57,036 INFO
    org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 1000 rows
    for join key [SIA285, WSSS-NZAA, 2013-08-09 22:47:00]
    2014-03-08 00:59:45,131 INFO org.apache.hadoop.mapred.Task: Communication
    exception: java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedReader.<init>(BufferedReader.java:80)
    at java.io.BufferedReader.<init>(BufferedReader.java:91)
    at
    org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:515)
    at
    org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:189)
    at
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
    at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:812)
    at org.apache.hadoop.mapred.Task.updateCounters(Task.java:834)
    at org.apache.hadoop.mapred.Task.access$500(Task.java:72)
    at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:661)
    at java.lang.Thread.run(Thread.java:662)

    2014-03-08 01:02:18,727 INFO org.apache.hadoop.mapred.Task: Communication
    exception: java.lang.OutOfMemoryError: Java heap space

    2014-03-08 01:02:18,728 ERROR
    org.apache.hadoop.security.UserGroupInformation:
    PriviledgedActionException as:user.name (auth:SIMPLE)
    cause:org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST:
    File exists
    2014-03-08 01:02:26,632 WARN org.apache.hadoop.mapred.Child: Error
    running child
    org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File
    exists
    at
    org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
    at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: EEXIST: File exists
    at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
    at
    org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
    ... 7 more
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception
    java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
    at sun.nio.ch.IOUtil.write(IOUtil.java:40)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
    at
    org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
    at
    org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153)
    at
    org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at
    org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:518)
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block
    BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403
    in pipeline xx.xx.xx.xx:50010, xx.xx.xx.xx:50010: bad datanode
    xx.xx.xx.xx:50010
    2014-03-08 01:06:19,453 INFO org.apache.hadoop.mapred.Task: Runnning
    cleanup for the task



    Regards,

    Jagan M

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Serega Sheypak at Mar 8, 2014 at 4:46 pm
    The other suggestion
    check your free space. Are you sure that you have enough non-HDFS space for
    intermediate output? Mapreduce have to store intermediate output (after map
    and before reduce) somewhere. Also reduce-side needs to fetch and merge map
    output. It uses non-HDFS space.


    2014-03-08 17:31 GMT+04:00 Jagan Madhappan <madhajagan@gmail.com>:
    /etc/hadoop/conf file has been updated .

    <property>
    <name>mapred.child.java.opts</name>
    <value> -Xmx1073741824</value>
    </property>

    In which property i have to change for increase reducer heap size
    using CM4.5.0 because my mappers were completed without any issue only
    reducers creating an issue

    Regards,
    Jagan
    On 3/8/14, Jagan Madhappan wrote:
    Darren ,
    Thanks for your response .
    My mappers completed successfully but i am facing issue in reduce phase .
    I have increased heap size using CM & changed below property
    Gateway --> Resource Management ->MapReduce Child Java Maximum Heap Size to
    1GB then restarted job tracker & ran deploy client configuration

    Regards,
    Jagan M

    On Saturday, March 8, 2014 10:01:51 AM UTC+5:30, Darren Lo wrote:

    Hi Jagan,

    It looks like the memory of the MapReduce job is insufficient. Which
    java
    heap size did you increase? I'd check your MapReduce Gateway
    configurations
    for the various heaps and make sure those are configured to have
    sufficient
    heap. Be sure to deploy client configurations to make sure your
    /etc/hadoop/conf picks up your changes.

    cdh-...@cloudera.org <javascript:> may be able to offer better tips.

    Thanks,
    Darren


    On Fri, Mar 7, 2014 at 6:15 PM, Jagan Madhappan
    <madha...@gmail.com<javascript:>
    wrote:
    *Hi Team ,*

    We are using CM4.5.0 [ CDH4.2 + Hive 11 ]
    I am getting below error message when i run big jobs using hive . I
    have
    increased the Java Heap size but its not helping me to fix this issue .
    Please do needful .


    *stderr logs*

    Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError:
    Java heap space
    at
    java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
    at
    java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
    at
    org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:158)
    Exception in thread "ResponseProcessor for block
    BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403"
    java.lang.OutOfMemoryError: Java heap space
    at java.util.HashMap.addEntry(HashMap.java:753)
    at java.util.HashMap.put(HashMap.java:385)
    at java.util.HashSet.add(HashSet.java:200)
    at
    sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:104)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:70)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at
    org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at
    org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
    at
    org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
    at
    org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:670)
    log4j:WARN No appenders could be found for logger
    (org.apache.hadoop.hdfs.DFSClient).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfigfor
    more info.

    ------------------------------


    *syslog logs*

    FO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100 rows
    2014-03-08 00:56:41,151 INFO
    org.apache.hadoop.hive.ql.exec.JoinOperator:
    5 forwarding 1000 rows
    2014-03-08 00:56:41,152 INFO
    org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows
    2014-03-08 00:56:54,609 INFO
    org.apache.hadoop.hive.ql.exec.JoinOperator:
    5 forwarding 10000 rows
    2014-03-08 00:56:54,609 INFO
    org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows
    2014-03-08 00:56:57,036 INFO
    org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 1000
    rows
    for join key [SIA285, WSSS-NZAA, 2013-08-09 22:47:00]
    2014-03-08 00:59:45,131 INFO org.apache.hadoop.mapred.Task:
    Communication
    exception: java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedReader.<init>(BufferedReader.java:80)
    at java.io.BufferedReader.<init>(BufferedReader.java:91)
    at
    org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:515)
    at
    org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:189)
    at
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
    at
    org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:812)
    at org.apache.hadoop.mapred.Task.updateCounters(Task.java:834)
    at org.apache.hadoop.mapred.Task.access$500(Task.java:72)
    at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:661)
    at java.lang.Thread.run(Thread.java:662)

    2014-03-08 01:02:18,727 INFO org.apache.hadoop.mapred.Task:
    Communication
    exception: java.lang.OutOfMemoryError: Java heap space

    2014-03-08 01:02:18,728 ERROR
    org.apache.hadoop.security.UserGroupInformation:
    PriviledgedActionException as:user.name (auth:SIMPLE)
    cause:org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException:
    EEXIST:
    File exists
    2014-03-08 01:02:26,632 WARN org.apache.hadoop.mapred.Child: Error
    running child
    org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST: File
    exists
    at
    org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
    at
    org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: EEXIST: File exists
    at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
    at
    org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
    ... 7 more
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception
    java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
    at sun.nio.ch.IOUtil.write(IOUtil.java:40)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
    at
    org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
    at
    org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153)
    at
    org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at
    java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at
    org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:518)
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block
    BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403
    in pipeline xx.xx.xx.xx:50010, xx.xx.xx.xx:50010: bad datanode
    xx.xx.xx.xx:50010
    2014-03-08 01:06:19,453 INFO org.apache.hadoop.mapred.Task: Runnning
    cleanup for the task



    Regards,

    Jagan M

    To unsubscribe from this group and stop receiving emails from it, send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.
  • Darren Lo at Mar 8, 2014 at 4:54 pm
    Hi Jagan,

    Try changing "Reduce Task Maximum Heap Size" in your MapReduce Gateway
    configuration, and re-deploy client configuration.

    Thanks,
    Darren

    On Sat, Mar 8, 2014 at 8:45 AM, Serega Sheypak wrote:

    The other suggestion
    check your free space. Are you sure that you have enough non-HDFS space
    for intermediate output? Mapreduce have to store intermediate output (after
    map and before reduce) somewhere. Also reduce-side needs to fetch and merge
    map output. It uses non-HDFS space.


    2014-03-08 17:31 GMT+04:00 Jagan Madhappan <madhajagan@gmail.com>:

    /etc/hadoop/conf file has been updated .
    <property>
    <name>mapred.child.java.opts</name>
    <value> -Xmx1073741824</value>
    </property>

    In which property i have to change for increase reducer heap size
    using CM4.5.0 because my mappers were completed without any issue only
    reducers creating an issue

    Regards,
    Jagan
    On 3/8/14, Jagan Madhappan wrote:
    Darren ,
    Thanks for your response .
    My mappers completed successfully but i am facing issue in reduce phase .
    I have increased heap size using CM & changed below property
    Gateway --> Resource Management ->MapReduce Child Java Maximum Heap Size to
    1GB then restarted job tracker & ran deploy client configuration

    Regards,
    Jagan M

    On Saturday, March 8, 2014 10:01:51 AM UTC+5:30, Darren Lo wrote:

    Hi Jagan,

    It looks like the memory of the MapReduce job is insufficient. Which
    java
    heap size did you increase? I'd check your MapReduce Gateway
    configurations
    for the various heaps and make sure those are configured to have
    sufficient
    heap. Be sure to deploy client configurations to make sure your
    /etc/hadoop/conf picks up your changes.

    cdh-...@cloudera.org <javascript:> may be able to offer better tips.

    Thanks,
    Darren


    On Fri, Mar 7, 2014 at 6:15 PM, Jagan Madhappan
    <madha...@gmail.com<javascript:>
    wrote:
    *Hi Team ,*

    We are using CM4.5.0 [ CDH4.2 + Hive 11 ]
    I am getting below error message when i run big jobs using hive . I
    have
    increased the Java Heap size but its not helping me to fix this issue
    .
    Please do needful .


    *stderr logs*

    Exception in thread "Thread for syncLogs" java.lang.OutOfMemoryError:
    Java heap space
    at
    java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59)
    at
    java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42)
    at
    org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:158)
    Exception in thread "ResponseProcessor for block
    BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403"
    java.lang.OutOfMemoryError: Java heap space
    at java.util.HashMap.addEntry(HashMap.java:753)
    at java.util.HashMap.put(HashMap.java:385)
    at java.util.HashSet.add(HashSet.java:200)
    at
    sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:104)
    at
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:70)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at
    org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:336)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
    at
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at java.io.FilterInputStream.read(FilterInputStream.java:66)
    at
    org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
    at
    org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
    at
    org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:670)
    log4j:WARN No appenders could be found for logger
    (org.apache.hadoop.hdfs.DFSClient).
    log4j:WARN Please initialize the log4j system properly.
    log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfigfor
    more info.

    ------------------------------


    *syslog logs*

    FO org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 100
    rows
    2014-03-08 00:56:41,151 INFO
    org.apache.hadoop.hive.ql.exec.JoinOperator:
    5 forwarding 1000 rows
    2014-03-08 00:56:41,152 INFO
    org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 1000 rows
    2014-03-08 00:56:54,609 INFO
    org.apache.hadoop.hive.ql.exec.JoinOperator:
    5 forwarding 10000 rows
    2014-03-08 00:56:54,609 INFO
    org.apache.hadoop.hive.ql.exec.SelectOperator: 6 forwarding 10000 rows
    2014-03-08 00:56:57,036 INFO
    org.apache.hadoop.hive.ql.exec.CommonJoinOperator: table 0 has 1000
    rows
    for join key [SIA285, WSSS-NZAA, 2013-08-09 22:47:00]
    2014-03-08 00:59:45,131 INFO org.apache.hadoop.mapred.Task:
    Communication
    exception: java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedReader.<init>(BufferedReader.java:80)
    at java.io.BufferedReader.<init>(BufferedReader.java:91)
    at
    org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:515)
    at
    org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:189)
    at
    org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
    at
    org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:812)
    at org.apache.hadoop.mapred.Task.updateCounters(Task.java:834)
    at org.apache.hadoop.mapred.Task.access$500(Task.java:72)
    at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:661)
    at java.lang.Thread.run(Thread.java:662)

    2014-03-08 01:02:18,727 INFO org.apache.hadoop.mapred.Task:
    Communication
    exception: java.lang.OutOfMemoryError: Java heap space

    2014-03-08 01:02:18,728 ERROR
    org.apache.hadoop.security.UserGroupInformation:
    PriviledgedActionException as:user.name (auth:SIMPLE)
    cause:org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException:
    EEXIST:
    File exists
    2014-03-08 01:02:26,632 WARN org.apache.hadoop.mapred.Child: Error
    running child
    org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException: EEXIST:
    File
    exists
    at
    org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:178)
    at
    org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:310)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:383)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: EEXIST: File exists
    at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
    at
    org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:172)
    ... 7 more
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient:
    DataStreamer Exception
    java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
    at sun.nio.ch.IOUtil.write(IOUtil.java:40)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
    at
    org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:62)
    at
    org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
    at
    org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:153)
    at
    org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:114)
    at
    java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at
    java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at
    org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:518)
    2014-03-08 01:02:49,187 WARN org.apache.hadoop.hdfs.DFSClient: Error
    Recovery for block
    BP-1597074705-xx.xx.xx.xx-1362569316393:blk_-2999799226967229671_3372403
    in pipeline xx.xx.xx.xx:50010, xx.xx.xx.xx:50010: bad datanode
    xx.xx.xx.xx:50010
    2014-03-08 01:06:19,453 INFO org.apache.hadoop.mapred.Task: Runnning
    cleanup for the task



    Regards,

    Jagan M

    To unsubscribe from this group and stop receiving emails from it,
    send
    an email to scm-users+...@cloudera.org <javascript:>.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an
    email to scm-users+unsubscribe@cloudera.org.
    To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+unsubscribe@cloudera.org.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMar 8, '14 at 2:15a
activeMar 8, '14 at 4:54p
posts6
users3
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase