Grokbase Groups Pig user October 2010
FAQ
Hi,

I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

Thanks,
-Rakesh

Search Discussions

  • Olga Natkovich at Oct 7, 2010 at 8:28 pm
    Not at this point as parameter substitution is implemented a s preprocessor on the script.

    Olga

    -----Original Message-----
    From: rakesh kothari
    Sent: Thursday, October 07, 2010 11:47 AM
    To: pig-user@hadoop.apache.org
    Subject: Passing parameters to Pig Script using Java


    Hi,

    I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

    Thanks,
    -Rakesh
  • Julien Le Dem at Oct 7, 2010 at 8:47 pm
    Here's my workaround:
    I extend PigServer with the following code copy/pasted from other places in Pig.
    Parameters are in the form: "foo=bar"
    Julien

    import org.apache.pig.ExecType;
    import org.apache.pig.backend.executionengine.ExecException;
    import org.apache.pig.backend.executionengine.ExecJob;
    import org.apache.pig.impl.PigContext;
    import org.apache.pig.tools.grunt.GruntParser;
    import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;


    /**
    * adapted from org.apache.pig.Main
    * returns the stream of final pig script to be passed to Grunt
    */
    private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
    List<String> paramFiles, String scriptFile)
    throws org.apache.pig.tools.parameters.ParseException, IOException{
    ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
    String[] type1 = new String[1];
    String[] type2 = new String[1];

    StringWriter writer = new StringWriter();
    psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer, params.size() > 0 ? params.toArray(type1) : null,
    paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);

    return new BufferedReader(new StringReader(writer.toString()));

    }

    /**
    * adapted original code from file to apply the preprocessor
    * @param filePath relative path of the pig script
    * @param params parameters to be applied by the preprocessor
    * @throws IOException
    */
    public void registerScript(String filePath, List<String> params) throws IOException {
    try {
    InputStream stream = new FileInputStream(filePath);
    try {
    GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
    grunt.setInteractive(false);
    grunt.setParams(this);
    grunt.parseStopOnError(true);
    } finally {
    try {
    stream.close();
    } catch (IOException e) {
    e.printStackTrace();
    }
    }
    } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
    throw new IOException("Error while parsing script: "+filePath,e);
    } catch (org.apache.pig.tools.parameters.ParseException e) {
    throw new IOException("Error while parsing parameters: "+params,e);
    }
    }

    /**
    * adapted original code from file to apply the preprocessor
    * @param filePath relative path of the pig script
    * @param params parameters to be applied by the preprocessor
    * @throws IOException
    */
    public void registerScript(String filePath, String... params) throws IOException {
    registerScript(filePath, Arrays.asList(params));
    }





    On 10/7/10 1:27 PM, "Olga Natkovich" wrote:

    Not at this point as parameter substitution is implemented a s preprocessor on the script.

    Olga

    -----Original Message-----
    From: rakesh kothari
    Sent: Thursday, October 07, 2010 11:47 AM
    To: pig-user@hadoop.apache.org
    Subject: Passing parameters to Pig Script using Java


    Hi,

    I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

    Thanks,
    -Rakesh
  • Jeff Zhang at Oct 8, 2010 at 1:23 am
    Hi Julien,

    You did what we have done in Pig 0.8. There's a little difference
    between your api and ours. We encapsulates the parameter in Map rather
    than List.

    On Fri, Oct 8, 2010 at 4:43 AM, Julien Le Dem wrote:
    Here's my workaround:
    I extend PigServer with the following code copy/pasted from other places in Pig.
    Parameters are in the form: "foo=bar"
    Julien

    import org.apache.pig.ExecType;
    import org.apache.pig.backend.executionengine.ExecException;
    import org.apache.pig.backend.executionengine.ExecJob;
    import org.apache.pig.impl.PigContext;
    import org.apache.pig.tools.grunt.GruntParser;
    import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;


    /**
    * adapted from org.apache.pig.Main
    * returns the stream of final pig script to be passed to Grunt
    */
    private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
    List<String> paramFiles, String scriptFile)
    throws org.apache.pig.tools.parameters.ParseException, IOException{
    ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
    String[] type1 = new String[1];
    String[] type2 = new String[1];

    StringWriter writer = new StringWriter();
    psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer,  params.size() > 0 ? params.toArray(type1) : null,
    paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);

    return new BufferedReader(new StringReader(writer.toString()));

    }

    /**
    * adapted original code from file to apply the preprocessor
    * @param filePath relative path of the pig script
    * @param params parameters to be applied by the preprocessor
    * @throws IOException
    */
    public void registerScript(String filePath, List<String> params) throws IOException {
    try {
    InputStream stream = new FileInputStream(filePath);
    try {
    GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
    grunt.setInteractive(false);
    grunt.setParams(this);
    grunt.parseStopOnError(true);
    } finally {
    try {
    stream.close();
    } catch (IOException e) {
    e.printStackTrace();
    }
    }
    } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
    throw new IOException("Error while parsing script: "+filePath,e);
    } catch (org.apache.pig.tools.parameters.ParseException e) {
    throw new IOException("Error while parsing parameters: "+params,e);
    }
    }

    /**
    * adapted original code from file to apply the preprocessor
    * @param filePath relative path of the pig script
    * @param params parameters to be applied by the preprocessor
    * @throws IOException
    */
    public void registerScript(String filePath, String... params) throws IOException {
    registerScript(filePath, Arrays.asList(params));
    }





    On 10/7/10 1:27 PM, "Olga Natkovich" wrote:

    Not at this point as parameter substitution is implemented a s preprocessor on the script.

    Olga

    -----Original Message-----
    From: rakesh kothari
    Sent: Thursday, October 07, 2010 11:47 AM
    To: pig-user@hadoop.apache.org
    Subject: Passing parameters to Pig Script using Java


    Hi,

    I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

    Thanks,
    -Rakesh



    --
    Best Regards

    Jeff Zhang
  • Julien Le Dem at Oct 8, 2010 at 5:05 pm
    Oops, sorry
    Yes this is old code.
    Thanks for pointing this out.
    This should give a hint though.
    Julien


    On 10/7/10 6:16 PM, "Jeff Zhang" wrote:

    Hi Julien,

    You did what we have done in Pig 0.8. There's a little difference
    between your api and ours. We encapsulates the parameter in Map rather
    than List.

    On Fri, Oct 8, 2010 at 4:43 AM, Julien Le Dem wrote:
    Here's my workaround:
    I extend PigServer with the following code copy/pasted from other places in Pig.
    Parameters are in the form: "foo=bar"
    Julien

    import org.apache.pig.ExecType;
    import org.apache.pig.backend.executionengine.ExecException;
    import org.apache.pig.backend.executionengine.ExecJob;
    import org.apache.pig.impl.PigContext;
    import org.apache.pig.tools.grunt.GruntParser;
    import org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor;


    /**
    * adapted from org.apache.pig.Main
    * returns the stream of final pig script to be passed to Grunt
    */
    private Reader runParamPreprocessor(InputStream origPigScript, List<String> params,
    List<String> paramFiles, String scriptFile)
    throws org.apache.pig.tools.parameters.ParseException, IOException{
    ParameterSubstitutionPreprocessor psp = new ParameterSubstitutionPreprocessor(50);
    String[] type1 = new String[1];
    String[] type2 = new String[1];

    StringWriter writer = new StringWriter();
    psp.genSubstitutedFile (new BufferedReader(new InputStreamReader(origPigScript)), writer, params.size() > 0 ? params.toArray(type1) : null,
    paramFiles.size() > 0 ? paramFiles.toArray(type2) : null);

    return new BufferedReader(new StringReader(writer.toString()));

    }

    /**
    * adapted original code from file to apply the preprocessor
    * @param filePath relative path of the pig script
    * @param params parameters to be applied by the preprocessor
    * @throws IOException
    */
    public void registerScript(String filePath, List<String> params) throws IOException {
    try {
    InputStream stream = new FileInputStream(filePath);
    try {
    GruntParser grunt = new GruntParser(runParamPreprocessor(stream, params, new ArrayList<String>(), filePath));
    grunt.setInteractive(false);
    grunt.setParams(this);
    grunt.parseStopOnError(true);
    } finally {
    try {
    stream.close();
    } catch (IOException e) {
    e.printStackTrace();
    }
    }
    } catch (org.apache.pig.tools.pigscript.parser.ParseException e) {
    throw new IOException("Error while parsing script: "+filePath,e);
    } catch (org.apache.pig.tools.parameters.ParseException e) {
    throw new IOException("Error while parsing parameters: "+params,e);
    }
    }

    /**
    * adapted original code from file to apply the preprocessor
    * @param filePath relative path of the pig script
    * @param params parameters to be applied by the preprocessor
    * @throws IOException
    */
    public void registerScript(String filePath, String... params) throws IOException {
    registerScript(filePath, Arrays.asList(params));
    }





    On 10/7/10 1:27 PM, "Olga Natkovich" wrote:

    Not at this point as parameter substitution is implemented a s preprocessor on the script.

    Olga

    -----Original Message-----
    From: rakesh kothari
    Sent: Thursday, October 07, 2010 11:47 AM
    To: pig-user@hadoop.apache.org
    Subject: Passing parameters to Pig Script using Java


    Hi,

    I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

    Thanks,
    -Rakesh



    --
    Best Regards

    Jeff Zhang
  • Gerrit van Vuuren at Oct 7, 2010 at 8:40 pm
    Hi,

    Have a look at the grunt source code, you'll need to pass your script through the preprocess stage with thr parameters to create a substituted pig file, which you can then pass to pig. I've written code that does this before by just doing what grunt does.

    Cheers,

    ----- Original Message -----
    From: Olga Natkovich <olgan@yahoo-inc.com>
    To: user@pig.apache.org <user@pig.apache.org>; pig-user@hadoop.apache.org <pig-user@hadoop.apache.org>
    Sent: Thu Oct 07 21:27:58 2010
    Subject: RE: Passing parameters to Pig Script using Java

    Not at this point as parameter substitution is implemented a s preprocessor on the script.

    Olga

    -----Original Message-----
    From: rakesh kothari
    Sent: Thursday, October 07, 2010 11:47 AM
    To: pig-user@hadoop.apache.org
    Subject: Passing parameters to Pig Script using Java


    Hi,

    I have a pig script that needs certain parameters (passed using "-p" in pig shell) to execute. Is there a way to pass these parameters if I want to execute this script using "PigServer" after registering the script using PigServer.registerScript() ?

    Thanks,
    -Rakesh

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedOct 7, '10 at 6:47p
activeOct 8, '10 at 5:05p
posts6
users5
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase