FAQ
Hi,

How do I change the default timeout for reducer with Pig? I have some
reducer that needs to take longer than 10 minutes to finish. It is pretty
frustrating to see many of get to 95% complete and then got killed.

Thanks.
Dexin

Search Discussions

  • Dmitriy Ryaboy at Dec 21, 2010 at 9:33 pm
    "set mapred.task.timeout $num_milliseconds"


    On Tue, Dec 21, 2010 at 1:23 PM, Dexin Wang wrote:

    Hi,

    How do I change the default timeout for reducer with Pig? I have some
    reducer that needs to take longer than 10 minutes to finish. It is pretty
    frustrating to see many of get to 95% complete and then got killed.

    Thanks.
    Dexin
  • Ashutosh Chauhan at Dec 21, 2010 at 9:55 pm
    Ideally you need not to do that. Pig automatically takes care of
    progress reporting in its operator. Do you have a pig script which
    fails because of reporting progress timeout issues ?

    Ashutosh
    On Tue, Dec 21, 2010 at 13:23, Dexin Wang wrote:
    Hi,

    How do I change the default timeout for reducer with Pig? I have some
    reducer that needs to take longer than 10 minutes to finish. It is pretty
    frustrating to see many of get to 95% complete and then got killed.

    Thanks.
    Dexin
  • Daniel Dai at Dec 21, 2010 at 10:40 pm
    True, however there is one bug in 0.7. We fix it in 0.8.

    https://issues.apache.org/jira/browse/PIG-1760

    Daniel


    Ashutosh Chauhan wrote:
    Ideally you need not to do that. Pig automatically takes care of
    progress reporting in its operator. Do you have a pig script which
    fails because of reporting progress timeout issues ?

    Ashutosh
    On Tue, Dec 21, 2010 at 13:23, Dexin Wang wrote:

    Hi,

    How do I change the default timeout for reducer with Pig? I have some
    reducer that needs to take longer than 10 minutes to finish. It is pretty
    frustrating to see many of get to 95% complete and then got killed.

    Thanks.
    Dexin
  • Dexin Wang at Dec 23, 2010 at 3:38 am
    It doesn't seem to work. I got

    2010-12-22 21:36:59,120 [main] ERROR org.apache.pig.tools.grunt.Grunt -
    ERROR 1000: Error during parsing. Unrecognized set key: mapred.task.timeout

    I did this in my pig script:

    SET mapred.task.timeout 1800000;

    What did I do wrong? Thanks.

    % pig --version
    Apache Pig version 0.7.0+9 (rexported)
    compiled Jun 28 2010, 12:53:50
    On Tue, Dec 21, 2010 at 2:39 PM, Daniel Dai wrote:

    True, however there is one bug in 0.7. We fix it in 0.8.

    https://issues.apache.org/jira/browse/PIG-1760

    Daniel



    Ashutosh Chauhan wrote:
    Ideally you need not to do that. Pig automatically takes care of
    progress reporting in its operator. Do you have a pig script which
    fails because of reporting progress timeout issues ?

    Ashutosh

    On Tue, Dec 21, 2010 at 13:23, Dexin Wang wrote:

    Hi,

    How do I change the default timeout for reducer with Pig? I have some
    reducer that needs to take longer than 10 minutes to finish. It is pretty
    frustrating to see many of get to 95% complete and then got killed.

    Thanks.
    Dexin

  • Ashutosh Chauhan at Dec 23, 2010 at 4:30 pm
    SET is only available in 0.8 not in 0.7. But I think your problem lies
    elsewhere and most likely is not a reporting timeout issue.

    Ashutosh
    On Wednesday, December 22, 2010, Dexin Wang wrote:
    It doesn't seem to work. I got

    2010-12-22 21:36:59,120 [main] ERROR org.apache.pig.tools.grunt.Grunt -
    ERROR 1000: Error during parsing. Unrecognized set key: mapred.task.timeout

    I did this in my pig script:

    SET mapred.task.timeout 1800000;

    What did I do wrong? Thanks.

    % pig --version
    Apache Pig version 0.7.0+9 (rexported)
    compiled Jun 28 2010, 12:53:50
    On Tue, Dec 21, 2010 at 2:39 PM, Daniel Dai wrote:

    True, however there is one bug in 0.7. We fix it in 0.8.

    https://issues.apache.org/jira/browse/PIG-1760

    Daniel



    Ashutosh Chauhan wrote:
    Ideally you need not to do that. Pig automatically takes care of
    progress reporting in its operator. Do you have a pig script which
    fails because of reporting progress timeout issues ?

    Ashutosh

    On Tue, Dec 21, 2010 at 13:23, Dexin Wang wrote:

    Hi,

    How do I change the default timeout for reducer with Pig? I have some
    reducer that needs to take longer than 10 minutes to finish. It is pretty
    frustrating to see many of get to 95% complete and then got killed.

    Thanks.
    Dexin

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupuser @
categoriespig, hadoop
postedDec 21, '10 at 9:23p
activeDec 23, '10 at 4:30p
posts6
users4
websitepig.apache.org

People

Translate

site design / logo © 2021 Grokbase