Grokbase Groups Pig user October 2010
FAQ
Hello,
I want to set more heap space to my scripts, but I can't make Pig
support this, when I call: "pig -Dmapred.child.java.opts=-Xmx2048"
It fails (just prints help), and option --help doesn't show any
information about -D parameter.

How can I do this?
I'm using version 0.7.0+9.
--
Wojciech Langiewicz

Search Discussions

  • Marcos Medrado Rubinelli at Oct 22, 2010 at 8:20 pm
    Wojcieh,

    One alternative is setting this and other parameters in a
    mapred-site.xml file in your classpath.

    Chhers,
    Marcos
    Hello,
    I want to set more heap space to my scripts, but I can't make Pig
    support this, when I call: "pig -Dmapred.child.java.opts=-Xmx2048"
    It fails (just prints help), and option --help doesn't show any
    information about -D parameter.

    How can I do this?
    I'm using version 0.7.0+9.
    --
    Wojciech Langiewicz
  • Wojciech Langiewicz at Oct 24, 2010 at 8:46 pm
    Hi,
    But will this work with different settings for every job?
    Because this is what I essentially need.

    Marcos Medrado Rubinelli wrote:
    Wojcieh,

    One alternative is setting this and other parameters in a
    mapred-site.xml file in your classpath.

    Chhers,
    Marcos
    Hello,
    I want to set more heap space to my scripts, but I can't make Pig
    support this, when I call: "pig -Dmapred.child.java.opts=-Xmx2048"
    It fails (just prints help), and option --help doesn't show any
    information about -D parameter.

    How can I do this?
    I'm using version 0.7.0+9.
    --
    Wojciech Langiewicz
  • Rekha Joshi at Oct 25, 2010 at 5:56 am
    Hi Wojciech,

    Well, the parameter mapred.child.java.opts is also subject to memory limit of your box and by the limit set by mapred.child.ulimit.
    It is ideal though to find the root cause of heap error. If needed use specific mapred.(map|reduce).child.java.opts as needed.

    In some cases it needs to be rectified by using nocombiner (-Dpig.exec.nocombiner=true) but I reckon that would be 0.8..

    Also as Olga confirmed from 0.8 you would be able to see -D params list settable from pig command using -help -properties

    Thanks & Regards,
    /Rekha

    On 10/25/10 2:15 AM, "Wojciech Langiewicz" wrote:

    Hi,
    But will this work with different settings for every job?
    Because this is what I essentially need.

    Marcos Medrado Rubinelli wrote:
    Wojcieh,

    One alternative is setting this and other parameters in a
    mapred-site.xml file in your classpath.

    Chhers,
    Marcos
    Hello,
    I want to set more heap space to my scripts, but I can't make Pig
    support this, when I call: "pig -Dmapred.child.java.opts=-Xmx2048"
    It fails (just prints help), and option --help doesn't show any
    information about -D parameter.

    How can I do this?
    I'm using version 0.7.0+9.
    --
    Wojciech Langiewicz
  • Ashutosh Chauhan at Oct 25, 2010 at 4:02 pm
    If you want to externally specify properties for *a* pig script you
    can put the key value pairs in a conf file and pass its location using
    --config to pig and those properties will be included.

    If you want to externally specify properties for *all* pig script that
    you run, then modify conf/pig.properties by putting in your key value
    pairs in it.

    If you want to specify properties from *within* a pig script then you
    can use set command in the pig script. This feature will be there in
    upcoming 0.8 release.

    Hope it helps,
    Ashutosh

    On Sun, Oct 24, 2010 at 13:45, Wojciech Langiewicz
    wrote:
    Hi,
    But will this work with different settings for every job?
    Because this is what I essentially need.

    Marcos Medrado Rubinelli wrote:
    Wojcieh,

    One alternative is setting this and other parameters in a
    mapred-site.xml file in your classpath.

    Chhers,
    Marcos
    Hello,
    I want to set more heap space to my scripts, but I can't make Pig
    support this, when I call: "pig -Dmapred.child.java.opts=-Xmx2048"
    It fails (just prints help), and option --help doesn't show any
    information about -D parameter.

    How can I do this?
    I'm using version 0.7.0+9.
    --
    Wojciech Langiewicz
  • Kochis, Allan at Oct 25, 2010 at 4:09 pm
    The way I get pig options is:

    PIG_OPTS="$PIG_OPTS
    -Dmapred.cache.archives=hdfs://server:8020/pat/data.$date#index "
    export PIG_OPTS


    pig ./Piggy.pig

    Allan

    -----Original Message-----
    From: Ashutosh Chauhan
    Sent: Sunday, October 24, 2010 6:43 PM
    To: user@pig.apache.org
    Subject: Re: Passing Java options to pig scripts

    If you want to externally specify properties for *a* pig script you can
    put the key value pairs in a conf file and pass its location using
    --config to pig and those properties will be included.

    If you want to externally specify properties for *all* pig script that
    you run, then modify conf/pig.properties by putting in your key value
    pairs in it.

    If you want to specify properties from *within* a pig script then you
    can use set command in the pig script. This feature will be there in
    upcoming 0.8 release.

    Hope it helps,
    Ashutosh

    On Sun, Oct 24, 2010 at 13:45, Wojciech Langiewicz
    wrote:
    Hi,
    But will this work with different settings for every job?
    Because this is what I essentially need.

    Marcos Medrado Rubinelli wrote:
    Wojcieh,

    One alternative is setting this and other parameters in a
    mapred-site.xml file in your classpath.

    Chhers,
    Marcos
    Hello,
    I want to set more heap space to my scripts, but I can't make Pig
    support this, when I call: "pig -Dmapred.child.java.opts=-Xmx2048"
    It fails (just prints help), and option --help doesn't show any
    information about -D parameter.

    How can I do this?
    I'm using version 0.7.0+9.
    --
    Wojciech Langiewicz

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedOct 22, '10 at 4:17p
activeOct 25, '10 at 4:09p
posts6
users5
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase