FAQ
hi all,

we have plan to runs parallel MR and break several job into particular
map or reduce to get ideal workload on hadoop cluster. we aren't
prefer to use scheduler cause depend on user, is it possible to
ovewrite MR jobconf via
Command line ? (e.g hadoop -jar test.jar -jobconf mapred.map.tasks=2 -
jobconf mapred.reduce.tasks=2 /test/input /test/output )



-mugi-

Search Discussions

  • Raj Vishwanathan at Mar 19, 2012 at 3:26 pm
    Mugi

    You can overwrite any job related configuration on the command line.

    For example

    hadoop -jar test.jar -Dmapred.reduce.tasks=2  /test/input /test/output

    Raj


    ________________________________
    From: mugi <i9um0p@gmail.com>
    To: CDH Users <cdh-user@cloudera.org>
    Sent: Monday, March 19, 2012 5:59 AM
    Subject: overwrite hadoop jobconf

    hi all,

    we have plan to runs parallel MR and break several job into particular
    map or  reduce  to get ideal workload on hadoop cluster. we aren't
    prefer to use scheduler cause depend on user, is it possible to
    ovewrite MR jobconf via
    Command line ? (e.g hadoop -jar test.jar -jobconf mapred.map.tasks=2 -
    jobconf mapred.reduce.tasks=2 /test/input /test/output )



    -mugi-


  • Mugi at Mar 22, 2012 at 1:08 pm
    raj,

    your command raise an error following bellow :

    hadoop jar /home/mapruser/Sample/workloadtest.jar -
    Dmapred.map.tasks=2 -Dmapred.reduce.tasks=1 workloadtest.run /test/
    isat /test/tstout1
    Exception in thread "main" java.lang.ClassNotFoundException: -
    Dmapred.map.tasks=2
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:179)

    then i change your command sequence to :
    hadoop jar /home/mapruser/Sample/workloadtest.jar workloadtest.run /
    test/isat /test/tstout1 -Dmapred.map.tasks=2 -Dmapred.reduce.tasks=1
    it runs, but MR job still using all map capacity so i can't control
    the workload.


    -mugi-

    On Mar 19, 10:26 pm, Raj Vishwanathan wrote:
    Mugi

    You can overwrite any job related configuration on the command line.

    For example

    hadoop -jar test.jar -Dmapred.reduce.tasks=2  /test/input /test/output

    Raj






    ________________________________
    From: mugi <i9u...@gmail.com>
    To: CDH Users <cdh-u...@cloudera.org>
    Sent: Monday, March 19, 2012 5:59 AM
    Subject: overwrite hadoop jobconf
    hi all,
    we have plan to runs parallel MR and break several job into particular
    map or  reduce  to get ideal workload on hadoop cluster. we aren't
    prefer to use scheduler cause depend on user, is it possible to
    ovewrite MR jobconf via
    Command line ? (e.g hadoop -jar test.jar -jobconf mapred.map.tasks=2 -
    jobconf mapred.reduce.tasks=2 /test/input /test/output )
    -mugi-
  • Harsh J at Mar 22, 2012 at 1:55 pm
    Mugi,

    Change your "fixed" command to:

    hadoop jar /home/mapruser/Sample/workloadtest.jar workloadtest.run
    -Dmapred.map.tasks=2 -Dmapred.reduce.tasks=1 /test/isat /test/tstout1

    The -D parameters for ToolRunner must always go first.
    On Thu, Mar 22, 2012 at 6:38 PM, mugi wrote:
    raj,

    your command raise an error following bellow :

    hadoop jar  /home/mapruser/Sample/workloadtest.jar -
    Dmapred.map.tasks=2 -Dmapred.reduce.tasks=1  workloadtest.run  /test/
    isat /test/tstout1
    Exception in thread "main" java.lang.ClassNotFoundException: -
    Dmapred.map.tasks=2
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:179)

    then i change your command sequence to :
    hadoop jar  /home/mapruser/Sample/workloadtest.jar  workloadtest.run  /
    test/isat /test/tstout1  -Dmapred.map.tasks=2 -Dmapred.reduce.tasks=1
    it runs,  but MR job still using all map capacity so i can't control
    the workload.


    -mugi-

    On Mar 19, 10:26 pm, Raj Vishwanathan wrote:
    Mugi

    You can overwrite any job related configuration on the command line.

    For example

    hadoop -jar test.jar -Dmapred.reduce.tasks=2  /test/input /test/output

    Raj






    ________________________________
    From: mugi <i9u...@gmail.com>
    To: CDH Users <cdh-u...@cloudera.org>
    Sent: Monday, March 19, 2012 5:59 AM
    Subject: overwrite hadoop jobconf
    hi all,
    we have plan to runs parallel MR and break several job into particular
    map or  reduce  to get ideal workload on hadoop cluster. we aren't
    prefer to use scheduler cause depend on user, is it possible to
    ovewrite MR jobconf via
    Command line ? (e.g hadoop -jar test.jar -jobconf mapred.map.tasks=2 -
    jobconf mapred.reduce.tasks=2 /test/input /test/output )
    -mugi-


    --
    Harsh J
  • Joey Echeverria at Mar 22, 2012 at 10:24 pm
    The expand a little bit on Harsh's description. The generic pattern is:

    hadoop jar <path to jar file> <class with main method> <ToolRunner
    arguments> <class specific arguments>

    -Joey
    On Thu, Mar 22, 2012 at 9:54 AM, Harsh J wrote:
    Mugi,

    Change your "fixed" command to:

    hadoop jar /home/mapruser/Sample/workloadtest.jar workloadtest.run
    -Dmapred.map.tasks=2 -Dmapred.reduce.tasks=1 /test/isat /test/tstout1

    The -D parameters for ToolRunner must always go first.
    On Thu, Mar 22, 2012 at 6:38 PM, mugi wrote:
    raj,

    your command raise an error following bellow :

    hadoop jar  /home/mapruser/Sample/workloadtest.jar -
    Dmapred.map.tasks=2 -Dmapred.reduce.tasks=1  workloadtest.run  /test/
    isat /test/tstout1
    Exception in thread "main" java.lang.ClassNotFoundException: -
    Dmapred.map.tasks=2
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:179)

    then i change your command sequence to :
    hadoop jar  /home/mapruser/Sample/workloadtest.jar  workloadtest.run  /
    test/isat /test/tstout1  -Dmapred.map.tasks=2 -Dmapred.reduce.tasks=1
    it runs,  but MR job still using all map capacity so i can't control
    the workload.


    -mugi-

    On Mar 19, 10:26 pm, Raj Vishwanathan wrote:
    Mugi

    You can overwrite any job related configuration on the command line.

    For example

    hadoop -jar test.jar -Dmapred.reduce.tasks=2  /test/input /test/output

    Raj






    ________________________________
    From: mugi <i9u...@gmail.com>
    To: CDH Users <cdh-u...@cloudera.org>
    Sent: Monday, March 19, 2012 5:59 AM
    Subject: overwrite hadoop jobconf
    hi all,
    we have plan to runs parallel MR and break several job into particular
    map or  reduce  to get ideal workload on hadoop cluster. we aren't
    prefer to use scheduler cause depend on user, is it possible to
    ovewrite MR jobconf via
    Command line ? (e.g hadoop -jar test.jar -jobconf mapred.map.tasks=2 -
    jobconf mapred.reduce.tasks=2 /test/input /test/output )
    -mugi-


    --
    Harsh J


    --
    Joey Echeverria
    Solutions Architect
    Cloudera, Inc.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupcdh-user @
categorieshadoop
postedMar 19, '12 at 12:59p
activeMar 22, '12 at 10:24p
posts5
users4
websitecloudera.com
irc#hadoop

People

Translate

site design / logo © 2022 Grokbase