Grokbase › Groups › Pig › dev › February 2008
FAQ
Impossible to set jobconf parameters
------------------------------------

Key: PIG-93
URL: https://issues.apache.org/jira/browse/PIG-93
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.1.0
Reporter: Benjamin Francisoud
Priority: Critical


I'm trying to set jobconf parameter before launching a pig job using pig api.

I tried 2 different ways but with no success:
{code:java}
PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
pigContext.getExecutionEngine().getConfiguration().putAll(properties);
PigServer pigServer = new PigServer(pigContext);
....
{code}

Throw a NPE because the internal executionEngine var is initialize only when calling connect().

So I tried:
{code:java}
PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
pigContext.connect();
pigContext.getExecutionEngine().getConfiguration().putAll(properties);
PigServer pigServer = new PigServer(pigContext);
...
{code}

My properties have been replace with a "new JobConf()"
{noformat}
java.lang.RuntimeException: Bad mapred.job.tracker: local
at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
{noformat}

"properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"

Before PIG-32 I use to do (and it was working):
{code:java}
PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
pigContext.setConf(myJobConf);
PigServer pigServer = new PigServer(pigContext);
...
{code}

Any idea before I start to work on a patch ?

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Benjamin Francisoud (JIRA) at Feb 5, 2008 at 4:28 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565794#action_12565794 ]

    Benjamin Francisoud commented on PIG-93:
    ----------------------------------------

    2 other ways that don't work:

    no connect():

    {code:java}
    final PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setJobtrackerLocation(properties.get("mapred.job.tracker"));
    pigContext.setJobName(properties.get("fs.default.name"));
    final PigServer pigServer = new PigServer(pigContext);
    {code}

    {noformat}
    java.lang.NullPointerException
    at org.apache.pig.impl.PigContext.setJobtrackerLocation(PigContext.java:213)
    ...
    {noformat}

    using connect():

    {code:java}
    final PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.setJobtrackerLocation(properties.get("mapred.job.tracker"));
    pigContext.setJobName(properties.get("fs.default.name"));
    final PigServer pigServer = new PigServer(pigContext);
    {code}

    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}

    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical

    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Antonio Magnaghi at Feb 5, 2008 at 5:57 pm ⇧
    Benjamin,

    Can you please attach a fragment of the code you were using previously
    and that was working ok?
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    I am using a copy from svn that does not contain the code factorization
    changes, but don't' see setConf on PigContext.

    Thanks,
    -a.

    -----Original Message-----
    From: Benjamin Francisoud (JIRA)
    Sent: Tuesday, February 05, 2008 8:20 AM
    To: [email protected]
    Subject: [jira] Created: (PIG-93) Impossible to set jobconf parameters

    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical


    I'm trying to set jobconf parameter before launching a pig job using pig
    api.

    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}

    Throw a NPE because the internal executionEngine var is initialize only
    when calling connect().

    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}

    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExe
    cutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}

    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"

    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}

    Any idea before I start to work on a patch ?

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Benjamin Francisoud at Feb 6, 2008 at 8:37 am ⇧

    Antonio Magnaghi a écrit :
    Benjamin,

    Can you please attach a fragment of the code you were using previously
    and that was working ok?
    Do you mean PigContext code ? or my own "client" code ?
    Because my own code is already under this line ;)
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    I am using a copy from svn that does not contain the code factorization
    changes, but don't' see setConf on PigContext.

    Thanks,
    -a.
    pigContext.setConf(myJobConf) method was (briefly) present in PigContext r616129 because of PIG-69
  • Antonio Magnaghi at Feb 5, 2008 at 6:44 pm ⇧
    If I understand correctly the problem, one possible way to configure the
    job conf before launching the job to a given cluster could be:

    PigContext pigContext = new PigContext(PigServer.ExecType.MAPREDUCE);
    PigServer pigServer = new PigServer(pigContext);

    Properties props = new Properties();

    //
    // set specific values in props...
    // ...

    // set these properties in the job conf. They will be present in the job
    conf used when MR is executed
    pigContext.getExecutionEngine().getConfiguration().putAll(props);

    // register queries with the Pig Server that will trigger MR jobs(s)


    -----Original Message-----
    From: Benjamin Francisoud (JIRA)
    Sent: Tuesday, February 05, 2008 8:20 AM
    To: [email protected]
    Subject: [jira] Created: (PIG-93) Impossible to set jobconf parameters

    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical


    I'm trying to set jobconf parameter before launching a pig job using pig
    api.

    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}

    Throw a NPE because the internal executionEngine var is initialize only
    when calling connect().

    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}

    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExe
    cutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}

    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"

    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}

    Any idea before I start to work on a patch ?

    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Benjamin Francisoud at Feb 6, 2008 at 8:49 am ⇧

    Antonio Magnaghi a écrit :
    If I understand correctly the problem, one possible way to configure the
    job conf before launching the job to a given cluster could be:

    PigContext pigContext = new PigContext(PigServer.ExecType.MAPREDUCE);
    PigServer pigServer = new PigServer(pigContext);

    Properties props = new Properties();

    //
    // set specific values in props...
    // ...

    // set these properties in the job conf. They will be present in the job
    conf used when MR is executed
    pigContext.getExecutionEngine().getConfiguration().putAll(props);

    // register queries with the Pig Server that will trigger MR jobs(s)
    Yes you understood correctly :)

    There is a slight difference between your example above and the ones I
    putted in PIG-93.
    You are creating the pigServer object using pigContext object before
    setting the properties in pigContext with putAll() method.

    But it doesn't seem to solve the problem:

    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    at org.apache.pig.PigServer.<init>(PigServer.java:123)
    -----Original Message-----
    From: Benjamin Francisoud (JIRA)
    Sent: Tuesday, February 05, 2008 8:20 AM
    To: [email protected]
    Subject: [jira] Created: (PIG-93) Impossible to set jobconf parameters

    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
  • Antonio Magnaghi at Feb 6, 2008 at 3:12 pm ⇧
    From the trace you have attached below it looks like the connection to the job tracker on the cluster failed, before getting to the point to set job specific properties and before launching a job to the cluster.
    I tried out what I suggested yesterday by setting some custom keys after creating a pig server and was able to see such keys in the job conf shipped to the cluster.

    Can you please attach a complete code sample that reproduces the problem so that I can reproduce the issue and further debug it?

    Thanks,
    -a.

    -----Original Message-----
    From: Benjamin Francisoud
    Sent: Wednesday, February 06, 2008 12:49 AM
    To: [email protected]
    Subject: Re: [jira] Created: (PIG-93) Impossible to set jobconf parameters

    Antonio Magnaghi a écrit :
    If I understand correctly the problem, one possible way to configure the
    job conf before launching the job to a given cluster could be:

    PigContext pigContext = new PigContext(PigServer.ExecType.MAPREDUCE);
    PigServer pigServer = new PigServer(pigContext);

    Properties props = new Properties();

    //
    // set specific values in props...
    // ...

    // set these properties in the job conf. They will be present in the job
    conf used when MR is executed
    pigContext.getExecutionEngine().getConfiguration().putAll(props);

    // register queries with the Pig Server that will trigger MR jobs(s)
    Yes you understood correctly :)

    There is a slight difference between your example above and the ones I
    putted in PIG-93.
    You are creating the pigServer object using pigContext object before
    setting the properties in pigContext with putAll() method.

    But it doesn't seem to solve the problem:

    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    at org.apache.pig.PigServer.<init>(PigServer.java:123)
    -----Original Message-----
    From: Benjamin Francisoud (JIRA)
    Sent: Tuesday, February 05, 2008 8:20 AM
    To: [email protected]
    Subject: [jira] Created: (PIG-93) Impossible to set jobconf parameters

    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
  • Benjamin Francisoud at Feb 6, 2008 at 3:26 pm ⇧
    Weird, I'll create a simple main file to repoduce it.

    Antonio Magnaghi a écrit :
    From the trace you have attached below it looks like the connection to the job tracker on the cluster failed, before getting to the point to set job specific properties and before launching a job to the cluster.

    I tried out what I suggested yesterday by setting some custom keys after creating a pig server and was able to see such keys in the job conf shipped to the cluster.

    Can you please attach a complete code sample that reproduces the problem so that I can reproduce the issue and further debug it?

    Thanks,
    -a.

    -----Original Message-----
    From: Benjamin Francisoud
    Sent: Wednesday, February 06, 2008 12:49 AM
    To: [email protected]
    Subject: Re: [jira] Created: (PIG-93) Impossible to set jobconf parameters

    Antonio Magnaghi a écrit :
    If I understand correctly the problem, one possible way to configure the
    job conf before launching the job to a given cluster could be:

    PigContext pigContext = new PigContext(PigServer.ExecType.MAPREDUCE);
    PigServer pigServer = new PigServer(pigContext);

    Properties props = new Properties();

    //
    // set specific values in props...
    // ...

    // set these properties in the job conf. They will be present in the job
    conf used when MR is executed
    pigContext.getExecutionEngine().getConfiguration().putAll(props);

    // register queries with the Pig Server that will trigger MR jobs(s)

    Yes you understood correctly :)

    There is a slight difference between your example above and the ones I
    putted in PIG-93.
    You are creating the pigServer object using pigContext object before
    setting the properties in pigContext with putAll() method.

    But it doesn't seem to solve the problem:

    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    at org.apache.pig.PigServer.<init>(PigServer.java:123)

    -----Original Message-----
    From: Benjamin Francisoud (JIRA)
    Sent: Tuesday, February 05, 2008 8:20 AM
    To: [email protected]
    Subject: [jira] Created: (PIG-93) Impossible to set jobconf parameters

    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
  • Benjamin Francisoud at Feb 6, 2008 at 5:17 pm ⇧
    Get the PIG93Main.java from https://issues.apache.org/jira/browse/PIG-93
    Put it in src/org/apache/pig
    Change FS_NAME, JOB_TRACKER, INPUT_PATH and OUTPUT_PATH values

    And you should get the same error as me.

    I'm using r619075 with no local modifcations.

    Benjamin Francisoud a écrit :
    Weird, I'll create a simple main file to repoduce it.

    Antonio Magnaghi a écrit :
    From the trace you have attached below it looks like the connection
    to the job tracker on the cluster failed, before getting to the point
    to set job specific properties and before launching a job to the
    cluster.

    I tried out what I suggested yesterday by setting some custom keys
    after creating a pig server and was able to see such keys in the job
    conf shipped to the cluster.

    Can you please attach a complete code sample that reproduces the
    problem so that I can reproduce the issue and further debug it?

    Thanks,
    -a.

    -----Original Message-----
    From: Benjamin Francisoud
    Sent: Wednesday, February 06, 2008 12:49 AM
    To: [email protected]
    Subject: Re: [jira] Created: (PIG-93) Impossible to set jobconf
    parameters

    Antonio Magnaghi a écrit :
    If I understand correctly the problem, one possible way to configure
    the
    job conf before launching the job to a given cluster could be:

    PigContext pigContext = new PigContext(PigServer.ExecType.MAPREDUCE);
    PigServer pigServer = new PigServer(pigContext);

    Properties props = new Properties();

    //
    // set specific values in props...
    // ...

    // set these properties in the job conf. They will be present in the
    job
    conf used when MR is executed
    pigContext.getExecutionEngine().getConfiguration().putAll(props);

    // register queries with the Pig Server that will trigger MR jobs(s)
    Yes you understood correctly :)

    There is a slight difference between your example above and the ones
    I putted in PIG-93.
    You are creating the pigServer object using pigContext object before
    setting the properties in pigContext with putAll() method.

    But it doesn't seem to solve the problem:

    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at
    org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)

    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    at org.apache.pig.PigServer.<init>(PigServer.java:123)

    -----Original Message-----
    From: Benjamin Francisoud (JIRA) Sent:
    Tuesday, February 05, 2008 8:20 AM
    To: [email protected]
    Subject: [jira] Created: (PIG-93) Impossible to set jobconf parameters

    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
  • Benjamin Francisoud (JIRA) at Feb 6, 2008 at 5:15 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Benjamin Francisoud updated PIG-93:
    -----------------------------------

    Attachment: PIG93Main.java

    Java Main to reproduce this bug.
    You need to change FS_NAME, JOB_TRACKER, INPUT_PATH and OUTPUT_PATH values.

    The stacktrace:
    {noformat}
    Running main...
    08/02/06 18:11:04 INFO apache.pig: Connecting to hadoop file system at: file:///
    08/02/06 18:11:05 INFO apache.pig: Connecting to map-reduce job tracker at: local
    Exception in thread "main" java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:151)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:182)
    at org.apache.pig.PigServer.(PIG93Main.java:22)
    {noformat}
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Feb 25, 2008 at 8:49 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-93:
    ------------------------------

    Patch Info: (was: [Patch Available])

    Cleared patch available flag since this is not a patch that needs to be committed
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Benjamin Francisoud (JIRA) at Feb 27, 2008 at 10:34 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12572860#action_12572860 ]

    Benjamin Francisoud commented on PIG-93:
    ----------------------------------------

    Yes you are right, I got caught by something else and didn't provide the patch rapidly...
    Started to work on that now ;)
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Benjamin Francisoud (JIRA) at Feb 28, 2008 at 4:26 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573344#action_12573344 ]

    Benjamin Francisoud commented on PIG-93:
    ----------------------------------------

    This issue is currently being fix in PIG-111.
    Will close it once PIG-111 patch gets commited in trunk.
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Eugene Kirpichov (JIRA) at Feb 29, 2008 at 10:25 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573665#action_12573665 ]

    Eugene Kirpichov commented on PIG-93:
    -------------------------------------

    I am having a similar problem while trying to get started with Pig.
    I've successfully installed Hadoop on 2 machines and hadoop's examples finally worked.
    Now, I built Pig and that's what I get when launching it:

    <code>
    jkff@***** trunk$ scripts/pig.pl -v -cp pig.jar:~/hadoop/hadoop-0.16.0/conf
    I can't find HOD configuration for , hopefully you weren't planning on using HOD.
    2008-02-29 13:19:22,102 [main] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config()
    at org.apache.hadoop.conf.Configuration.(JobConf.java:112)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.(PigContext.java:179)
    at org.apache.pig.PigServer.(Grunt.java:41)
    at org.apache.pig.Main.main(Main.java:247)

    2008-02-29 13:19:22,259 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
    2008-02-29 13:19:22,263 [main] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config()
    at org.apache.hadoop.conf.Configuration.(HConfiguration.java:38)
    at org.apache.pig.backend.hadoop.datastorage.HDataStorage.(HExecutionEngine.java:140)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:181)
    at org.apache.pig.PigServer.(Grunt.java:41)
    at org.apache.pig.Main.main(Main.java:247)

    2008-02-29 13:19:22,418 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: local
    2008-02-29 13:19:22,428 [main] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config()
    at org.apache.hadoop.conf.Configuration.(HConfiguration.java:38)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:181)
    at org.apache.pig.PigServer.(Grunt.java:41)
    at org.apache.pig.Main.main(Main.java:247)

    2008-02-29 13:19:22,610 [main] ERROR org.apache.pig.Main - java.lang.RuntimeException: Bad mapred.job.tracker: local
    </code>

    I've been grepping the sources for a couple of hours already to find out where these properties really should have been set to something senseful, but I found that the HExecutionEngine constructor is actually invoked in its 1-argument version, where it creates an 'empty' JobConf() and does not seem to be going to read my hadoop-site.conf.

    I am pretty sure I made a mistake somewhere, because it can't be that pig simply doesn't work at all from command line, and I'd be much pleased to be pointed at this mistake :)

    It would be even better if you had a from-scratch description like 'How to install pig and hadoop and launch a simple example on two machines' in the wiki.
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Benjamin Francisoud (JIRA) at Feb 29, 2008 at 10:53 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573672#action_12573672 ]

    Benjamin Francisoud commented on PIG-93:
    ----------------------------------------

    Eugene you should watch (and vote for) PIG-111, I tested the latest patch and it fixes PIG-93.

    I like the idea of making an example page in the wiki for that, I will take care of it once the new PigServer and PigContext api is commited in svn...
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Eugene Kirpichov (JIRA) at Feb 29, 2008 at 11:38 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573690#action_12573690 ]

    Eugene Kirpichov commented on PIG-93:
    -------------------------------------

    Thanks, I'll have a look there.
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at May 19, 2008 at 6:08 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598033#action_12598033 ]

    Olga Natkovich commented on PIG-93:
    -----------------------------------

    PIG-111 has been committed a while back. Can this issue be closed?
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Pi Song (JIRA) at May 20, 2008 at 9:05 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-93?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Pi Song resolved PIG-93.
    ------------------------

    Resolution: Fixed

    Already fixed as a part of Pig-111
    Impossible to set jobconf parameters
    ------------------------------------

    Key: PIG-93
    URL: https://issues.apache.org/jira/browse/PIG-93
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.1.0
    Reporter: Benjamin Francisoud
    Priority: Critical
    Attachments: PIG93Main.java


    I'm trying to set jobconf parameter before launching a pig job using pig api.
    I tried 2 different ways but with no success:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ....
    {code}
    Throw a NPE because the internal executionEngine var is initialize only when calling connect().
    So I tried:
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.connect();
    pigContext.getExecutionEngine().getConfiguration().putAll(properties);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    My properties have been replace with a "new JobConf()"
    {noformat}
    java.lang.RuntimeException: Bad mapred.job.tracker: local
    at org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:711)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:149)
    at org.apache.pig.impl.PigContext.connect(PigContext.java:180)
    {noformat}
    "properties" contains "mapred.job.tracker" and "hadoop.tmp.dir values"
    Before PIG-32 I use to do (and it was working):
    {code:java}
    PigContext pigContext = new PigContext(ExecType.MAPREDUCE);
    pigContext.setConf(myJobConf);
    PigServer pigServer = new PigServer(pigContext);
    ...
    {code}
    Any idea before I start to work on a patch ?
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedFeb 5, '08 at 4:20p
activeMay 20, '08 at 9:05a
posts18
users3
websitepig.apache.org

3 users in discussion

Pi Song (JIRA): 11 posts Benjamin Francisoud: 4 posts Antonio Magnaghi: 3 posts

Content

People

Support

Translate

site design / logo © 2023 Grokbase