FAQ
Hi Experts,

I an running my 4 node hadoop cluster using CMS 4.5. when I submit the pig
job it is giving alots of warning can any one have any solution of it.

this is the log of the run:

$ pig event.pig
2013-04-02 18:01:12,776 [main] INFO org.apache.pig.Main - Apache Pig
version 0.10.0-cdh4.2.0 (rexported) compiled Feb 15 2013, 12:19:02
2013-04-02 18:01:12,776 [main] INFO org.apache.pig.Main - Logging error
messages to: /home/hadoopoc/pigScript/pig_1364896872772.log
2013-04-02 18:01:13,277 [main] INFO org.apache.pig.impl.util.Utils -
Default bootup file /home/hadoopoc/.pigbootup not found
2013-04-02 18:01:13,560 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: hdfs://master:8020
2013-04-02 18:01:13,564 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-04-02 18:01:15,040 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to map-reduce job tracker at: master:8021
2013-04-02 18:01:15,042 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.df.interval is deprecated. Instead, use fs.df.interval
2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration -
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration -
fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.block.size is deprecated. Instead, use dfs.blocksize
2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.access.time.precision is deprecated. Instead, use
dfs.namenode.accesstime.precision
2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.replication.considerLoad is deprecated. Instead, use
dfs.namenode.replication.considerLoad
2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.balance.bandwidthPerSec is deprecated. Instead, use
dfs.datanode.balance.bandwidthPerSec
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.safemode.threshold.pct is deprecated. Instead, use
dfs.namenode.safemode.threshold-pct
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.name.dir.restore is deprecated. Instead, use
dfs.namenode.name.dir.restore
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.https.client.keystore.resource is deprecated. Instead, use
dfs.client.https.keystore.resource
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.backup.http.address is deprecated. Instead, use
dfs.namenode.backup.http-address
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.safemode.extension is deprecated. Instead, use
dfs.namenode.safemode.extension
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.datanode.max.xcievers is deprecated. Instead, use
dfs.datanode.max.transfer.threads
2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.https.need.client.auth is deprecated. Instead, use
dfs.client.https.need-auth
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.replication.interval is deprecated. Instead, use
dfs.namenode.replication.interval
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
fs.checkpoint.edits.dir is deprecated. Instead, use
dfs.namenode.checkpoint.edits.dir
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.write.packet.size is deprecated. Instead, use
dfs.client-write-packet-size
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.permissions.supergroup is deprecated. Instead, use
dfs.permissions.superusergroup
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
topology.script.number.args is deprecated. Instead, use
net.topology.script.number.args
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.umaskmode is deprecated. Instead, use fs.permissions.umask-mode
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
dfs.secondary.http.address is deprecated. Instead, use
dfs.namenode.secondary.http-address
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
fs.checkpoint.period is deprecated. Instead, use
dfs.namenode.checkpoint.period
2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration -
topology.node.switch.mapping.impl is deprecated. Instead, use
net.topology.node.switch.mapping.impl
2013-04-02 18:01:15,186 [main] WARN org.apache.hadoop.conf.Configuration -
io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-04-02 18:01:16,318 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: LIMIT
2013-04-02 18:01:17,305 [main] INFO
org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned
for record: $0, $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13,
$14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27
2013-04-02 18:01:17,650 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2013-04-02 18:01:17,740 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 2
2013-04-02 18:01:17,740 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 2
2013-04-02 18:01:17,969 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2013-04-02 18:01:17,989 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-04-02 18:01:17,991 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting Parallelism to 1
2013-04-02 18:01:18,252 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job7191830890692043765.jar
2013-04-02 18:01:23,975 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job7191830890692043765.jar created
2013-04-02 18:01:23,994 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2013-04-02 18:01:24,017 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false,
will not generate code.
2013-04-02 18:01:24,017 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Starting process to move
generated code to distributed cacche
2013-04-02 18:01:24,017 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Setting key
[pig.schematuple.classes] with classes to deserialize []
2013-04-02 18:01:24,157 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2013-04-02 18:01:24,190 [JobControl] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
2013-04-02 18:01:24,598 [JobControl] WARN
org.apache.hadoop.conf.Configuration - fs.default.name is deprecated.
Instead, use fs.defaultFS
2013-04-02 18:01:24,598 [JobControl] WARN
org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated.
Instead, use dfs.namenode.https-address
2013-04-02 18:01:24,599 [JobControl] WARN
org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is
deprecated. Instead, use dfs.bytes-per-checksum
2013-04-02 18:01:24,666 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2013-04-02 18:01:24,691 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2013-04-02 18:01:24,738 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths (combined) to process : 1
2013-04-02 18:01:25,801 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201303221534_0111
2013-04-02 18:01:25,802 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases record,top_5
2013-04-02 18:01:25,802 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: record[3,9],null[-1,-1],top_5[41,8] C: R:
2013-04-02 18:01:25,802 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://master:50030/jobdetails.jsp?jobid=job_201303221534_0111
2013-04-02 18:01:35,135 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 25% complete
2013-04-02 18:01:45,114 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 50% complete
2013-04-02 18:01:50,835 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2013-04-02 18:01:50,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-04-02 18:01:50,836 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting Parallelism to 1
2013-04-02 18:01:50,859 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job2182491268421150083.jar
2013-04-02 18:01:55,232 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job2182491268421150083.jar created
2013-04-02 18:01:55,241 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2013-04-02 18:01:55,241 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false,
will not generate code.
2013-04-02 18:01:55,242 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Starting process to move
generated code to distributed cacche
2013-04-02 18:01:55,242 [main] INFO
org.apache.pig.data.SchemaTupleFrontend - Setting key
[pig.schematuple.classes] with classes to deserialize []
2013-04-02 18:01:55,277 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2013-04-02 18:01:55,301 [JobControl] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
2013-04-02 18:01:55,630 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2013-04-02 18:01:55,630 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1
2013-04-02 18:01:55,632 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths (combined) to process : 1
2013-04-02 18:01:56,418 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201303221534_0112
2013-04-02 18:01:56,418 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases group_bynow
2013-04-02 18:01:56,418 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: C: R: group_bynow[-1,-1]
2013-04-02 18:01:56,418 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://master:50030/jobdetails.jsp?jobid=job_201303221534_0112
2013-04-02 18:02:04,522 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 75% complete
2013-04-02 18:02:26,219 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2013-04-02 18:02:26,221 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
2.0.0-cdh4.2.0 0.10.0-cdh4.2.0 hadoopoc 2013-04-02 18:01:17
2013-04-02 18:02:26 LIMIT

Success!

Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime
MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime
MedianReducetime Alias Feature Outputs
job_201303221534_0111 1 1 4 4 4 4 8
8 8 8 record,top_5
job_201303221534_0112 1 1 3 3 3 3 8
8 8 8 group_bynow
hdfs://master:8020/tmp/temp2131406838/tmp516565134,

Input(s):
Successfully read 5 records (65922 bytes) from:
"hdfs://master:8020/user/hadoopoc/flume/data"

Output(s):
Successfully stored 5 records (81 bytes) in:
"hdfs://master:8020/tmp/temp2131406838/tmp516565134"

Counters:
Total records written : 5
Total bytes written : 81
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_201303221534_0111 -> job_201303221534_0112,
job_201303221534_0112


2013-04-02 18:02:26,239 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Success!
2013-04-02 18:02:26,242 [main] INFO org.apache.pig.data.SchemaTupleBackend
- Key [pig.schematuple] was not set... will not generate code.
2013-04-02 18:02:26,247 [main] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2013-04-02 18:02:26,247 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
input paths to process : 1
(null,1)
(null,1)
(null,1)
(null,null)
(null,null)

Search Discussions

  • Darren Lo at Apr 2, 2013 at 5:05 pm
    Hi Saurav,

    These deprecation warnings are a known issue with pig and can be safely
    ignored.

    Thanks,
    Darren

    On Tue, Apr 2, 2013 at 3:21 AM, Saurav Sinha wrote:

    Hi Experts,

    I an running my 4 node hadoop cluster using CMS 4.5. when I submit the pig
    job it is giving alots of warning can any one have any solution of it.

    this is the log of the run:

    $ pig event.pig
    2013-04-02 18:01:12,776 [main] INFO org.apache.pig.Main - Apache Pig
    version 0.10.0-cdh4.2.0 (rexported) compiled Feb 15 2013, 12:19:02
    2013-04-02 18:01:12,776 [main] INFO org.apache.pig.Main - Logging error
    messages to: /home/hadoopoc/pigScript/pig_1364896872772.log
    2013-04-02 18:01:13,277 [main] INFO org.apache.pig.impl.util.Utils -
    Default bootup file /home/hadoopoc/.pigbootup not found
    2013-04-02 18:01:13,560 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting to hadoop file system at: hdfs://master:8020
    2013-04-02 18:01:13,564 [main] WARN org.apache.hadoop.conf.Configuration
    - fs.default.name is deprecated. Instead, use fs.defaultFS
    2013-04-02 18:01:15,040 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
    Connecting to map-reduce job tracker at: master:8021
    2013-04-02 18:01:15,042 [main] WARN org.apache.hadoop.conf.Configuration
    - fs.default.name is deprecated. Instead, use fs.defaultFS
    2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.df.interval is deprecated. Instead, use fs.df.interval
    2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
    2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration
    - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
    2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
    2013-04-02 18:01:15,182 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
    2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration
    - fs.default.name is deprecated. Instead, use fs.defaultFS
    2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration
    - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
    2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.block.size is deprecated. Instead, use dfs.blocksize
    2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.access.time.precision is deprecated. Instead, use
    dfs.namenode.accesstime.precision
    2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.replication.min is deprecated. Instead, use
    dfs.namenode.replication.min
    2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
    2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.replication.considerLoad is deprecated. Instead, use
    dfs.namenode.replication.considerLoad
    2013-04-02 18:01:15,183 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.balance.bandwidthPerSec is deprecated. Instead, use
    dfs.datanode.balance.bandwidthPerSec
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.safemode.threshold.pct is deprecated. Instead, use
    dfs.namenode.safemode.threshold-pct
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.name.dir.restore is deprecated. Instead, use
    dfs.namenode.name.dir.restore
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.https.client.keystore.resource is deprecated. Instead, use
    dfs.client.https.keystore.resource
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.backup.http.address is deprecated. Instead, use
    dfs.namenode.backup.http-address
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.safemode.extension is deprecated. Instead, use
    dfs.namenode.safemode.extension
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.datanode.max.xcievers is deprecated. Instead, use
    dfs.datanode.max.transfer.threads
    2013-04-02 18:01:15,184 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.https.need.client.auth is deprecated. Instead, use
    dfs.client.https.need-auth
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.replication.interval is deprecated. Instead, use
    dfs.namenode.replication.interval
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - fs.checkpoint.edits.dir is deprecated. Instead, use
    dfs.namenode.checkpoint.edits.dir
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.write.packet.size is deprecated. Instead, use
    dfs.client-write-packet-size
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.permissions.supergroup is deprecated. Instead, use
    dfs.permissions.superusergroup
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - topology.script.number.args is deprecated. Instead, use
    net.topology.script.number.args
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.umaskmode is deprecated. Instead, use fs.permissions.umask-mode
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - dfs.secondary.http.address is deprecated. Instead, use
    dfs.namenode.secondary.http-address
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - fs.checkpoint.period is deprecated. Instead, use
    dfs.namenode.checkpoint.period
    2013-04-02 18:01:15,185 [main] WARN org.apache.hadoop.conf.Configuration
    - topology.node.switch.mapping.impl is deprecated. Instead, use
    net.topology.node.switch.mapping.impl
    2013-04-02 18:01:15,186 [main] WARN org.apache.hadoop.conf.Configuration
    - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
    2013-04-02 18:01:16,318 [main] INFO
    org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
    script: LIMIT
    2013-04-02 18:01:17,305 [main] INFO
    org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned
    for record: $0, $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13,
    $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27
    2013-04-02 18:01:17,650 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
    File concatenation threshold: 100 optimistic? false
    2013-04-02 18:01:17,740 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
    - MR plan size before optimization: 2
    2013-04-02 18:01:17,740 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
    - MR plan size after optimization: 2
    2013-04-02 18:01:17,969 [main] INFO
    org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
    to the job
    2013-04-02 18:01:17,989 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    2013-04-02 18:01:17,991 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - Setting Parallelism to 1
    2013-04-02 18:01:18,252 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - creating jar file Job7191830890692043765.jar
    2013-04-02 18:01:23,975 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - jar file Job7191830890692043765.jar created
    2013-04-02 18:01:23,994 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - Setting up single store job
    2013-04-02 18:01:24,017 [main] INFO
    org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false,
    will not generate code.
    2013-04-02 18:01:24,017 [main] INFO
    org.apache.pig.data.SchemaTupleFrontend - Starting process to move
    generated code to distributed cacche
    2013-04-02 18:01:24,017 [main] INFO
    org.apache.pig.data.SchemaTupleFrontend - Setting key
    [pig.schematuple.classes] with classes to deserialize []
    2013-04-02 18:01:24,157 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 1 map-reduce job(s) waiting for submission.
    2013-04-02 18:01:24,190 [JobControl] WARN
    org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
    the arguments. Applications should implement Tool for the same.
    2013-04-02 18:01:24,598 [JobControl] WARN
    org.apache.hadoop.conf.Configuration - fs.default.name is deprecated.
    Instead, use fs.defaultFS
    2013-04-02 18:01:24,598 [JobControl] WARN
    org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated.
    Instead, use dfs.namenode.https-address
    2013-04-02 18:01:24,599 [JobControl] WARN
    org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is
    deprecated. Instead, use dfs.bytes-per-checksum
    2013-04-02 18:01:24,666 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 0% complete
    2013-04-02 18:01:24,691 [JobControl] INFO
    org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
    to process : 1
    2013-04-02 18:01:24,738 [JobControl] INFO
    org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
    input paths (combined) to process : 1
    2013-04-02 18:01:25,801 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - HadoopJobId: job_201303221534_0111
    2013-04-02 18:01:25,802 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - Processing aliases record,top_5
    2013-04-02 18:01:25,802 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - detailed locations: M: record[3,9],null[-1,-1],top_5[41,8] C: R:
    2013-04-02 18:01:25,802 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - More information at:
    http://master:50030/jobdetails.jsp?jobid=job_201303221534_0111
    2013-04-02 18:01:35,135 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 25% complete
    2013-04-02 18:01:45,114 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 50% complete
    2013-04-02 18:01:50,835 [main] INFO
    org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
    to the job
    2013-04-02 18:01:50,835 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
    2013-04-02 18:01:50,836 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - Setting Parallelism to 1
    2013-04-02 18:01:50,859 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - creating jar file Job2182491268421150083.jar
    2013-04-02 18:01:55,232 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - jar file Job2182491268421150083.jar created
    2013-04-02 18:01:55,241 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
    - Setting up single store job
    2013-04-02 18:01:55,241 [main] INFO
    org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false,
    will not generate code.
    2013-04-02 18:01:55,242 [main] INFO
    org.apache.pig.data.SchemaTupleFrontend - Starting process to move
    generated code to distributed cacche
    2013-04-02 18:01:55,242 [main] INFO
    org.apache.pig.data.SchemaTupleFrontend - Setting key
    [pig.schematuple.classes] with classes to deserialize []
    2013-04-02 18:01:55,277 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 1 map-reduce job(s) waiting for submission.
    2013-04-02 18:01:55,301 [JobControl] WARN
    org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
    the arguments. Applications should implement Tool for the same.
    2013-04-02 18:01:55,630 [JobControl] INFO
    org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
    to process : 1
    2013-04-02 18:01:55,630 [JobControl] INFO
    org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
    input paths to process : 1
    2013-04-02 18:01:55,632 [JobControl] INFO
    org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
    input paths (combined) to process : 1
    2013-04-02 18:01:56,418 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - HadoopJobId: job_201303221534_0112
    2013-04-02 18:01:56,418 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - Processing aliases group_bynow
    2013-04-02 18:01:56,418 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - detailed locations: M: C: R: group_bynow[-1,-1]
    2013-04-02 18:01:56,418 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - More information at:
    http://master:50030/jobdetails.jsp?jobid=job_201303221534_0112
    2013-04-02 18:02:04,522 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 75% complete
    2013-04-02 18:02:26,219 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - 100% complete
    2013-04-02 18:02:26,221 [main] INFO
    org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

    HadoopVersion PigVersion UserId StartedAt FinishedAt
    Features
    2.0.0-cdh4.2.0 0.10.0-cdh4.2.0 hadoopoc 2013-04-02 18:01:17
    2013-04-02 18:02:26 LIMIT

    Success!

    Job Stats (time in seconds):
    JobId Maps Reduces MaxMapTime MinMapTIme AvgMapTime
    MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime
    MedianReducetime Alias Feature Outputs
    job_201303221534_0111 1 1 4 4 4 4 8
    8 8 8 record,top_5
    job_201303221534_0112 1 1 3 3 3 3 8
    8 8 8 group_bynow
    hdfs://master:8020/tmp/temp2131406838/tmp516565134,

    Input(s):
    Successfully read 5 records (65922 bytes) from:
    "hdfs://master:8020/user/hadoopoc/flume/data"

    Output(s):
    Successfully stored 5 records (81 bytes) in:
    "hdfs://master:8020/tmp/temp2131406838/tmp516565134"

    Counters:
    Total records written : 5
    Total bytes written : 81
    Spillable Memory Manager spill count : 0
    Total bags proactively spilled: 0
    Total records proactively spilled: 0

    Job DAG:
    job_201303221534_0111 -> job_201303221534_0112,
    job_201303221534_0112


    2013-04-02 18:02:26,239 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
    - Success!
    2013-04-02 18:02:26,242 [main] INFO
    org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not
    set... will not generate code.
    2013-04-02 18:02:26,247 [main] INFO
    org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
    to process : 1
    2013-04-02 18:02:26,247 [main] INFO
    org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
    input paths to process : 1
    (null,1)
    (null,1)
    (null,1)
    (null,null)
    (null,null)

    --
    Thanks,
    Darren

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupscm-users @
categorieshadoop
postedApr 2, '13 at 10:21a
activeApr 2, '13 at 5:05p
posts2
users2
websitecloudera.com
irc#hadoop

2 users in discussion

Saurav Sinha: 1 post Darren Lo: 1 post

People

Translate

site design / logo © 2022 Grokbase