FAQ
Dear all,


I've migrated my cluster to CM 4.5 free. There were some problems with
installations but I solved it.


Now some of my PIG scripts fails with very strange error. There are no
errors in my old cluster (CM Free 4.1.2)!
Here the screen messages:
--------------------
2013-03-31 12:35:47,493 [main] INFO org.apache.pig.Main - Apache Pig
version 0.10.0-cdh4.2.0 (rexported) compiled Feb 15 2013, 12:20:13
2013-03-31 12:35:47,493 [main] INFO org.apache.pig.Main - Logging
error messages to: /home/suggest/coldstart/pig_1364726147490.log
2013-03-31 12:35:47,745 [main] INFO org.apache.pig.impl.util.Utils -
Default bootup file /home/suggest/.pigbootup not found
2013-03-31 12:35:47,827 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: hdfs://hadoop3.rranalytics:8020
2013-03-31 12:35:47,830 [main] WARN
org.apache.hadoop.conf.Configuration - fs.default.name is deprecated.
Instead, use fs.defaultFS
2013-03-31 12:35:48,113 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to map-reduce job tracker at: hadoop3.rranalytics:8021
2013-03-31 12:35:48,114 [main] WARN
org.apache.hadoop.conf.Configuration - fs.default.name is deprecated.
Instead, use fs.defaultFS
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is
deprecated. Instead, use dfs.bytes-per-checksum
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.permissions.supergroup is
deprecated. Instead, use dfs.permissions.superusergroup
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.max.objects is deprecated.
Instead, use dfs.namenode.max.objects
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.replication.interval is
deprecated. Instead, use dfs.namenode.replication.interval
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.data.dir is deprecated.
Instead, use dfs.datanode.data.dir
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.access.time.precision is
deprecated. Instead, use dfs.namenode.accesstime.precision
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.replication.min is
deprecated. Instead, use dfs.namenode.replication.min
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - fs.checkpoint.dir is
deprecated. Instead, use dfs.namenode.checkpoint.dir
2013-03-31 12:35:48,492 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.http.address is deprecated.
Instead, use dfs.namenode.http-address
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.replication.considerLoad is
deprecated. Instead, use dfs.namenode.replication.considerLoad
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.write.packet.size is
deprecated. Instead, use dfs.client-write-packet-size
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.permissions is deprecated.
Instead, use dfs.permissions.enabled
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.block.size is deprecated.
Instead, use dfs.blocksize
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.https.address is
deprecated. Instead, use dfs.namenode.https-address
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.name.dir.restore is
deprecated. Instead, use dfs.namenode.name.dir.restore
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.https.need.client.auth is
deprecated. Instead, use dfs.client.https.need-auth
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.umaskmode is deprecated.
Instead, use fs.permissions.umask-mode
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration -
topology.node.switch.mapping.impl is deprecated. Instead, use
net.topology.node.switch.mapping.impl
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.backup.http.address is
deprecated. Instead, use dfs.namenode.backup.http-address
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.secondary.http.address is
deprecated. Instead, use dfs.namenode.secondary.http-address
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.safemode.extension is
deprecated. Instead, use dfs.namenode.safemode.extension
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.df.interval is deprecated.
Instead, use fs.df.interval
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - fs.checkpoint.edits.dir is
deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration -
dfs.https.client.keystore.resource is deprecated. Instead, use
dfs.client.https.keystore.resource
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.datanode.max.xcievers is
deprecated. Instead, use dfs.datanode.max.transfer.threads
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.backup.address is
deprecated. Instead, use dfs.namenode.backup.address
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - topology.script.number.args is
deprecated. Instead, use net.topology.script.number.args
2013-03-31 12:35:48,493 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.balance.bandwidthPerSec is
deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
2013-03-31 12:35:48,494 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.name.edits.dir is
deprecated. Instead, use dfs.namenode.edits.dir
2013-03-31 12:35:48,494 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.safemode.threshold.pct is
deprecated. Instead, use dfs.namenode.safemode.threshold-pct
2013-03-31 12:35:48,494 [main] WARN
org.apache.hadoop.conf.Configuration - dfs.name.dir is deprecated.
Instead, use dfs.namenode.name.dir
2013-03-31 12:35:48,496 [main] WARN
org.apache.hadoop.conf.Configuration - fs.checkpoint.period is
deprecated. Instead, use dfs.namenode.checkpoint.period
2013-03-31 12:35:48,496 [main] WARN
org.apache.hadoop.conf.Configuration - hadoop.native.lib is
deprecated. Instead, use io.native.lib.available
2013-03-31 12:35:51,350 [main] WARN org.apache.pig.PigServer -
Encountered Warning IMPLICIT_CAST_TO_LONG 9 time(s).
2013-03-31 12:35:51,350 [main] WARN org.apache.pig.PigServer -
Encountered Warning IMPLICIT_CAST_TO_FLOAT 4 time(s).
2013-03-31 12:36:12,862 [Low Memory Detector] INFO
org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
call- Usage threshold init = 172359680(168320K) used =
593734536(579818K) committed = 838860800(819200K) max =
838860800(819200K)
2013-03-31 12:36:22,162 [Low Memory Detector] INFO
org.apache.pig.impl.util.SpillableMemoryManager - first memory handler
call - Collection threshold init = 172359680(168320K) used =
566787568(553503K) committed = 838860800(819200K) max =
838860800(819200K)
2013-03-31 12:37:00,167 [main] ERROR org.apache.pig.tools.grunt.Grunt
- ERROR 2998: Unhandled internal error. Java heap space
2013-03-31 12:37:00,168 [main] ERROR org.apache.pig.tools.grunt.Grunt
- java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:
100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:
390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.shiftStringByTabs(LogicalPlanPrinter.java:
223)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
108)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirstLP(LogicalPlanPrinter.java:
83)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.visit(LogicalPlanPrinter.java:
69)
at
org.apache.pig.newplan.logical.relational.LogicalPlan.getSignature(LogicalPlan.java:
122)
at org.apache.pig.PigServer.execute(PigServer.java:1237)
at org.apache.pig.PigServer.access$400(PigServer.java:121)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1553)
at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:
991)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:
412)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:
194)

------------------
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. Java heap space

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:
100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:
390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.shiftStringByTabs(LogicalPlanPrinter.java:
223)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
108)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirst(LogicalPlanPrinter.java:
102)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirstLP(LogicalPlanPrinter.java:
83)
at
org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.visit(LogicalPlanPrinter.java:
69)
at
org.apache.pig.newplan.logical.relational.LogicalPlan.getSignature(LogicalPlan.java:
122)
at org.apache.pig.PigServer.execute(PigServer.java:1237)
at org.apache.pig.PigServer.access$400(PigServer.java:121)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1553)
at org.apache.pig.PigServer.registerQuery(PigServer.java:516)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:
991)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:
412)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:
194)
================================================================================

Search Discussions

  • Rzykov at Mar 31, 2013 at 2:05 pm
    I solved the problem by coping /usr/lib/pig from old to the new cluster

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedMar 31, '13 at 7:42a
activeMar 31, '13 at 2:05p
posts2
users1
websitecloudera.com
irc#hadoop

1 user in discussion

Rzykov: 2 posts

People

Translate

site design / logo © 2022 Grokbase