Grokbase Groups Pig dev March 2011
FAQ
NativeMapReduce does not Allow Configuration Parameters containing Spaces
--------------------------------------------------------------------------

Key: PIG-1917
URL: https://issues.apache.org/jira/browse/PIG-1917
Project: Pig
Issue Type: Bug
Reporter: Mitesh Singh Jat


In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

Considering a simple pig script with ''wordcount'' as Native M-R job.

{code:title=nativeMR.pig}
A = load 'input/WordCountInput.txt';
B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
store B into 'output/WordCount' using PigStorage;
{code}

The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
'' -Xss128k"'' as the first argument to '''wordcount'''.
Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.

Physical plan, Logical plan, and M-R plan of pig show double quotes.
For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
{monospace}
MapReduce node scope-13
hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
--------
{monospace}

On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
{monospace}
$ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
...
...

{monospace}

Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Mitesh Singh Jat (JIRA) at Mar 18, 2011 at 9:51 am
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mitesh Singh Jat updated PIG-1917:
    ----------------------------------

    Description:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with ''wordcount'' as Native M-R job.

    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
    '' -Xss128k"'' as the first argument to '''wordcount'''.
    Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    {code}

    Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.

    was:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with ''wordcount'' as Native M-R job.

    {code:title=nativeMR.pig}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
    '' -Xss128k"'' as the first argument to '''wordcount'''.
    Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {monospace}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {monospace}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {monospace}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    {monospace}

    Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.

    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat

    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with ''wordcount'' as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
    '' -Xss128k"'' as the first argument to '''wordcount'''.
    Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Mitesh Singh Jat (JIRA) at Mar 18, 2011 at 9:53 am
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mitesh Singh Jat updated PIG-1917:
    ----------------------------------

    Description:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with ''wordcount'' as Native M-R job.

    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after ''-Dmapred.child.java.opts="-Xmx1536m'' makes
    ''-Xss128k"'' as the first argument to '''wordcount'''.
    Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    {code}

    Whereas, when ''-Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.

    was:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with ''wordcount'' as Native M-R job.

    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after '' -Dmapred.child.java.opts="-Xmx1536m'' makes
    '' -Xss128k"'' as the first argument to '''wordcount'''.
    Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    {code}

    Whereas, when '' -Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.

    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat

    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with ''wordcount'' as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after ''-Dmapred.child.java.opts="-Xmx1536m'' makes
    ''-Xss128k"'' as the first argument to '''wordcount'''.
    Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when ''-Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Mitesh Singh Jat (JIRA) at Mar 18, 2011 at 9:59 am
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mitesh Singh Jat updated PIG-1917:
    ----------------------------------

    Description:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with _wordcount_ as Native M-R job.

    {code:pig|title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {{
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    }}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {{
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    }}

    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.

    was:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with ''wordcount'' as Native M-R job.

    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after ''-Dmapred.child.java.opts="-Xmx1536m'' makes
    ''-Xss128k"'' as the first argument to '''wordcount'''.
    Even with other example M-R program, the parameter ''mapred.child.java.opts'' was assigned ''"-Xmx1536m'' in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    {code}

    Whereas, when ''-Dmapred.child.java.opts=-Xmx1536m'' was used, the *nativeMR.pig* executed successfully.

    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat

    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:pig|title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {{
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    }}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {{
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    }}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Mitesh Singh Jat (JIRA) at Mar 18, 2011 at 10:01 am
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mitesh Singh Jat updated PIG-1917:
    ----------------------------------

    Description:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with _wordcount_ as Native M-R job.

    {code:pig|title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    {code}

    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.

    was:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with _wordcount_ as Native M-R job.

    {code:pig|title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {{
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    }}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {{
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    }}

    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.

    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat

    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:pig|title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Mitesh Singh Jat (JIRA) at Mar 18, 2011 at 10:03 am
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Mitesh Singh Jat updated PIG-1917:
    ----------------------------------

    Description:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with _wordcount_ as Native M-R job.

    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    {code}

    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.

    was:
    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.

    Considering a simple pig script with _wordcount_ as Native M-R job.

    {code:pig|title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}

    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.

    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}

    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...

    {code}

    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.

    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat

    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Olga Natkovich (JIRA) at Mar 18, 2011 at 6:36 pm
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-1917:
    --------------------------------

    Fix Version/s: 0.8.0
    Assignee: Thejas M Nair
    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat
    Assignee: Thejas M Nair
    Fix For: 0.8.0


    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Thejas M Nair (JIRA) at Apr 7, 2011 at 11:06 pm
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017197#comment-13017197 ]

    Thejas M Nair commented on PIG-1917:
    ------------------------------------

    The pig parser is parsing the argument "-Dmapred.child.java.opts='-Xmx1536m -Xss128k'" to native mapreduce as two arguments "-Dmapred.child.java.opts='-Xmx1536m" and "-Xss128k'" .

    To support this use case, we need to parse it as one argument and remove the single quotes, similar to what bash shell does - "-Dmapred.child.java.opts=-Xmx1536m -Xss128k" (note the missing single quotes).

    In general, quoted arguments should be unquoted before being given as argument to native-mapreduce or streaming command, to simulate what the shell does. This would be a change in behavior that is not backward compatible. But it is a problem only if users rely on the delimiter quotes being included in the argument.

    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat
    Assignee: Thejas M Nair
    Fix For: 0.8.0


    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Thejas M Nair (JIRA) at Apr 8, 2011 at 12:35 am
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017219#comment-13017219 ]

    Thejas M Nair commented on PIG-1917:
    ------------------------------------

    To avoid breaking backward compatibility while supporting this use case, one option would be to have a way to switch to the mode that supports this use case.

    Maybe by adding a " using 'unquote-mode' " to the command ?

    eg .
    {code}
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int)
    `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir` using 'unquote-mode';
    {code}

    or maybe by setting a system property on commandline -
    -Dpig.unquote.cmdstring='true'
    I think adding a system property is better, as it does not pollute the syntax.


    In 0.9, maybe we can make the unquote mode the default mode.


    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat
    Assignee: Thejas M Nair
    Fix For: 0.8.0


    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Thejas M Nair (JIRA) at Apr 8, 2011 at 9:54 pm
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Thejas M Nair updated PIG-1917:
    -------------------------------

    Fix Version/s: (was: 0.8.0)
    0.9.0

    The change to parse "-Dmapred.child.java.opts='-Xmx1536m -Xss128k'" as one argument is also going to break backward compatibility, if any existing pig script is (accidentally) relying on that behavior. So making this change only for 0.9.
    Also, instead of unquoting all arguments, only the "-D" property arguments will be unquoted. This change will only affect Native map-reduce statements (ie, not streaming command).

    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat
    Assignee: Thejas M Nair
    Fix For: 0.9.0


    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Thejas M Nair (JIRA) at Apr 11, 2011 at 2:11 pm
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Thejas M Nair updated PIG-1917:
    -------------------------------

    Attachment: PIG-1917.1.patch
    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat
    Assignee: Thejas M Nair
    Fix For: 0.9.0

    Attachments: PIG-1917.1.patch


    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Richard Ding (JIRA) at Apr 12, 2011 at 9:20 pm
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019054#comment-13019054 ]

    Richard Ding commented on PIG-1917:
    -----------------------------------

    +1
    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat
    Assignee: Thejas M Nair
    Fix For: 0.9.0

    Attachments: PIG-1917.1.patch


    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Thejas M Nair (JIRA) at Apr 12, 2011 at 10:32 pm
    [ https://issues.apache.org/jira/browse/PIG-1917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Thejas M Nair resolved PIG-1917.
    --------------------------------

    Resolution: Fixed

    Unit test and test-patch passed.
    Patch committed to trunk.

    NativeMapReduce does not Allow Configuration Parameters containing Spaces
    --------------------------------------------------------------------------

    Key: PIG-1917
    URL: https://issues.apache.org/jira/browse/PIG-1917
    Project: Pig
    Issue Type: Bug
    Reporter: Mitesh Singh Jat
    Assignee: Thejas M Nair
    Fix For: 0.9.0

    Attachments: PIG-1917.1.patch


    In Pig, NativeMapReduce does not allow Configuration parameters which contain spaces.
    Considering a simple pig script with _wordcount_ as Native M-R job.
    {code:title=nativeMR.pig|borderStyle=solid}
    A = load 'input/WordCountInput.txt';
    B = MAPREDUCE 'hadoop-examples.jar' Store A into 'inputDir' Load 'outputDir' as (word:chararray, count:int) `wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir`;
    store B into 'output/WordCount' using PigStorage;
    {code}
    The above pig script fails while launching native MAPREDUCE job. Because space after _-Dmapred.child.java.opts="-Xmx1536m_ makes
    _-Xss128k"_ as the first argument to _wordcount_
    Even with other example M-R program, the parameter _mapred.child.java.opts_ was assigned _"-Xmx1536m_ in Jobconf.
    Physical plan, Logical plan, and M-R plan of pig show double quotes.
    For example, here is the corresponding M-R Plan for the native M-R "MAPREDUCE" statement.
    {code}
    MapReduce node scope-13
    hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir
    --------
    {code}
    On executing above M-R job after copying 'input/WordCountInput.txt' into 'inputDir/', as Hadoop Job, this ran successfully.
    {code}
    $ *hadoop jar hadoop-examples.jar wordcount -Dmapred.job.map.memory.mb=3072 -Dmapred.child.java.opts="-Xmx1536m -Xss128k" inputDir outputDir*
    11/03/18 14:23:29 INFO input.FileInputFormat: Total input paths to process : 1
    11/03/18 14:23:29 INFO mapred.JobClient: Running job: job_201103181353_0007
    11/03/18 14:23:30 INFO mapred.JobClient: map 0% reduce 0%
    11/03/18 14:23:37 INFO mapred.JobClient: map 100% reduce 0%
    11/03/18 14:23:49 INFO mapred.JobClient: map 100% reduce 100%
    11/03/18 14:23:51 INFO mapred.JobClient: Job complete: job_201103181353_0007
    ...
    ...
    {code}
    Whereas, when _-Dmapred.child.java.opts=-Xmx1536m_ was used, the *nativeMR.pig* executed successfully.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedMar 18, '11 at 9:49a
activeApr 12, '11 at 10:32p
posts13
users1
websitepig.apache.org

1 user in discussion

Thejas M Nair (JIRA): 13 posts

People

Translate

site design / logo © 2022 Grokbase