FAQ
Hi,

Could anyone please guide me as to how to use the -libjars option in HDFS?

I have added the necessary jar file (the hbase jar - to be precise) to the classpath of the node where I am starting the job.

The following is the format that i am invoking:
bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by Commas)> <Arguments to our main class>

bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a

Despite this, I find that I get the java.lang.ClassNotFoundException error! :(
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

The strange thing is that there is another MR job I have that runs perfectly with the libjars option! Could anybody tell me what I am doing wrong? One more thing - not sure if it is relevant : I am using the new Hadoop MapReduce API.

Thanks in advance!

Regards,
Krishnakumar.

Search Discussions

  • Ted Yu at Jan 11, 2011 at 6:24 pm
    Refer to Alex Kozlov's answer on 12/11/10

    On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
    wrote:
    Hi,

    Could anyone please guide me as to how to use the -libjars option in HDFS?

    I have added the necessary jar file (the hbase jar - to be precise) to the
    classpath of the node where I am starting the job.

    The following is the format that i am invoking:
    bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by
    Commas)> <Arguments to our main class>

    bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
    /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a

    Despite this, I find that I get the java.lang.ClassNotFoundException
    error! :(
    java.lang.RuntimeException: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
    at
    org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
    at
    org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

    The strange thing is that there is another MR job I have that runs
    perfectly with the libjars option! Could anybody tell me what I am doing
    wrong? One more thing - not sure if it is relevant : I am using the new
    Hadoop MapReduce API.

    Thanks in advance!

    Regards,
    Krishnakumar.
  • C.V.Krishnakumar Iyer at Jan 11, 2011 at 7:49 pm
    Hi,

    I have tried that as well, using -files <jar file> But it still gives the exact same error. Any other thing that I could try?

    Thanks,
    Krishna.
    On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:

    Refer to Alex Kozlov's answer on 12/11/10

    On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
    wrote:
    Hi,

    Could anyone please guide me as to how to use the -libjars option in HDFS?

    I have added the necessary jar file (the hbase jar - to be precise) to the
    classpath of the node where I am starting the job.

    The following is the format that i am invoking:
    bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated by
    Commas)> <Arguments to our main class>

    bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
    /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a

    Despite this, I find that I get the java.lang.ClassNotFoundException
    error! :(
    java.lang.RuntimeException: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
    at
    org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
    at
    org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

    The strange thing is that there is another MR job I have that runs
    perfectly with the libjars option! Could anybody tell me what I am doing
    wrong? One more thing - not sure if it is relevant : I am using the new
    Hadoop MapReduce API.

    Thanks in advance!

    Regards,
    Krishnakumar.
  • Alex Kozlov at Jan 11, 2011 at 8:34 pm
    Have you implemented GenericOptionsParser? Do you see your jar in the *
    mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
    via a JT Web UI)?

    --
    Alex Kozlov
    Solutions Architect
    Cloudera, Inc
    twitter: alexvk2009
    <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>


    On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer
    wrote:
    Hi,

    I have tried that as well, using -files <jar file> But it still gives the
    exact same error. Any other thing that I could try?

    Thanks,
    Krishna.
    On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:

    Refer to Alex Kozlov's answer on 12/11/10

    On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
    wrote:
    Hi,

    Could anyone please guide me as to how to use the -libjars option in
    HDFS?
    I have added the necessary jar file (the hbase jar - to be precise) to
    the
    classpath of the node where I am starting the job.

    The following is the format that i am invoking:
    bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars (separated
    by
    Commas)> <Arguments to our main class>

    bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
    /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a

    Despite this, I find that I get the java.lang.ClassNotFoundException
    error! :(
    java.lang.RuntimeException: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
    at
    org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
    at
    org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

    The strange thing is that there is another MR job I have that runs
    perfectly with the libjars option! Could anybody tell me what I am doing
    wrong? One more thing - not sure if it is relevant : I am using the new
    Hadoop MapReduce API.

    Thanks in advance!

    Regards,
    Krishnakumar.
  • Alex Kozlov at Jan 11, 2011 at 8:48 pm
    There is also a blog that I recently wrote, if it helps
    http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
    On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov wrote:

    Have you implemented GenericOptionsParser? Do you see your jar in the *
    mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
    via a JT Web UI)?

    --
    Alex Kozlov
    Solutions Architect
    Cloudera, Inc
    twitter: alexvk2009
    <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>


    On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
    f2004403@gmail.com> wrote:
    Hi,

    I have tried that as well, using -files <jar file> But it still gives the
    exact same error. Any other thing that I could try?

    Thanks,
    Krishna.
    On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:

    Refer to Alex Kozlov's answer on 12/11/10

    On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
    wrote:
    Hi,

    Could anyone please guide me as to how to use the -libjars option in
    HDFS?
    I have added the necessary jar file (the hbase jar - to be precise) to
    the
    classpath of the node where I am starting the job.

    The following is the format that i am invoking:
    bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
    (separated by
    Commas)> <Arguments to our main class>

    bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
    /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a

    Despite this, I find that I get the java.lang.ClassNotFoundException
    error! :(
    java.lang.RuntimeException: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
    at
    org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
    at
    org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
    at
    org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

    The strange thing is that there is another MR job I have that runs
    perfectly with the libjars option! Could anybody tell me what I am
    doing
    wrong? One more thing - not sure if it is relevant : I am using the new
    Hadoop MapReduce API.

    Thanks in advance!

    Regards,
    Krishnakumar.
  • C.V.Krishnakumar Iyer at Jan 11, 2011 at 10:54 pm
    Hi,

    Thanks a lot! I shall try this once and let you know!

    Regards,
    Krishna.
    On Jan 11, 2011, at 12:48 PM, Alex Kozlov wrote:

    There is also a blog that I recently wrote, if it helps
    http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
    On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov wrote:

    Have you implemented GenericOptionsParser? Do you see your jar in the *
    mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
    via a JT Web UI)?

    --
    Alex Kozlov
    Solutions Architect
    Cloudera, Inc
    twitter: alexvk2009
    <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>


    On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
    f2004403@gmail.com> wrote:
    Hi,

    I have tried that as well, using -files <jar file> But it still gives the
    exact same error. Any other thing that I could try?

    Thanks,
    Krishna.
    On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:

    Refer to Alex Kozlov's answer on 12/11/10

    On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
    wrote:
    Hi,

    Could anyone please guide me as to how to use the -libjars option in
    HDFS?
    I have added the necessary jar file (the hbase jar - to be precise) to
    the
    classpath of the node where I am starting the job.

    The following is the format that i am invoking:
    bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
    (separated by
    Commas)> <Arguments to our main class>

    bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
    /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a

    Despite this, I find that I get the java.lang.ClassNotFoundException
    error! :(
    java.lang.RuntimeException: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
    at
    org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
    at
    org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
    at
    org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

    The strange thing is that there is another MR job I have that runs
    perfectly with the libjars option! Could anybody tell me what I am
    doing
    wrong? One more thing - not sure if it is relevant : I am using the new
    Hadoop MapReduce API.

    Thanks in advance!

    Regards,
    Krishnakumar.
  • C.V.Krishnakumar Iyer at Jan 11, 2011 at 11:37 pm
    Hi,

    Thanks a lot Alex! using GenericOptionsParser solved the issue. Previously I had used Tool and had assumed that it would take care of this.

    Regards,
    Krishna.
    On Jan 11, 2011, at 12:48 PM, Alex Kozlov wrote:

    There is also a blog that I recently wrote, if it helps
    http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
    On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov wrote:

    Have you implemented GenericOptionsParser? Do you see your jar in the *
    mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view
    via a JT Web UI)?

    --
    Alex Kozlov
    Solutions Architect
    Cloudera, Inc
    twitter: alexvk2009
    <http://www.cloudera.com/company/press-center/hadoop-world-nyc/>


    On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer <
    f2004403@gmail.com> wrote:
    Hi,

    I have tried that as well, using -files <jar file> But it still gives the
    exact same error. Any other thing that I could try?

    Thanks,
    Krishna.
    On Jan 11, 2011, at 10:23 AM, Ted Yu wrote:

    Refer to Alex Kozlov's answer on 12/11/10

    On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer
    wrote:
    Hi,

    Could anyone please guide me as to how to use the -libjars option in
    HDFS?
    I have added the necessary jar file (the hbase jar - to be precise) to
    the
    classpath of the node where I am starting the job.

    The following is the format that i am invoking:
    bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars
    (separated by
    Commas)> <Arguments to our main class>

    bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars
    /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a

    Despite this, I find that I get the java.lang.ClassNotFoundException
    error! :(
    java.lang.RuntimeException: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841)
    at
    org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551)
    at
    org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793)
    at
    org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524)
    at
    org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
    Caused by: java.lang.RuntimeException:
    java.lang.ClassNotFoundException:
    org.apache.hadoop.hbase.io.ImmutableBytesWritable
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809)
    at
    org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833)

    The strange thing is that there is another MR job I have that runs
    perfectly with the libjars option! Could anybody tell me what I am
    doing
    wrong? One more thing - not sure if it is relevant : I am using the new
    Hadoop MapReduce API.

    Thanks in advance!

    Regards,
    Krishnakumar.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcommon-user @
categorieshadoop
postedJan 11, '11 at 6:11p
activeJan 11, '11 at 11:37p
posts7
users3
websitehadoop.apache.org...
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase