Grokbase Groups Pig dev January 2011
FAQ
mapred.output.compress in SET statement does not work
-----------------------------------------------------

Key: PIG-1814
URL: https://issues.apache.org/jira/browse/PIG-1814
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.8.0


Setting output compression using "SET" in the script does not work:
SET mapred.output.compress true;
SET mapred.output.compression.codec org.apache.hadoop.io.compress.GzipCodec;

We did some trick to make individual compression setting for multistore work. Instead of the above parameter, using the following works:
SET output.compression.enabled true;
SET output.compression.codec org.apache.hadoop.io.compress.GzipCodec;

However, this is against intuition. We should use mapred.output.compress/mapred.output.compression.codec.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Olga Natkovich (JIRA) at Mar 23, 2011 at 3:21 pm
    [ https://issues.apache.org/jira/browse/PIG-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-1814:
    --------------------------------

    Fix Version/s: (was: 0.8.0)
    0.9.0

    This issue has a very reasonable workaround - delaying till 0.9
    mapred.output.compress in SET statement does not work
    -----------------------------------------------------

    Key: PIG-1814
    URL: https://issues.apache.org/jira/browse/PIG-1814
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Daniel Dai
    Fix For: 0.9.0


    Setting output compression using "SET" in the script does not work:
    SET mapred.output.compress true;
    SET mapred.output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    We did some trick to make individual compression setting for multistore work. Instead of the above parameter, using the following works:
    SET output.compression.enabled true;
    SET output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    However, this is against intuition. We should use mapred.output.compress/mapred.output.compression.codec.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Daniel Dai (JIRA) at Apr 21, 2011 at 8:39 pm
    [ https://issues.apache.org/jira/browse/PIG-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Daniel Dai updated PIG-1814:
    ----------------------------

    Attachment: PIG-1814-1.patch
    mapred.output.compress in SET statement does not work
    -----------------------------------------------------

    Key: PIG-1814
    URL: https://issues.apache.org/jira/browse/PIG-1814
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Daniel Dai
    Fix For: 0.9.0

    Attachments: PIG-1814-1.patch


    Setting output compression using "SET" in the script does not work:
    SET mapred.output.compress true;
    SET mapred.output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    We did some trick to make individual compression setting for multistore work. Instead of the above parameter, using the following works:
    SET output.compression.enabled true;
    SET output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    However, this is against intuition. We should use mapred.output.compress/mapred.output.compression.codec.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • jiraposter@reviews.apache.org (JIRA) at Apr 25, 2011 at 5:13 pm
    [ https://issues.apache.org/jira/browse/PIG-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024853#comment-13024853 ]

    jiraposter@reviews.apache.org commented on PIG-1814:
    ----------------------------------------------------


    -----------------------------------------------------------
    This is an automatically generated e-mail. To reply, visit:
    https://reviews.apache.org/r/661/
    -----------------------------------------------------------

    Review request for pig.


    Summary
    -------

    See PIG-1814


    This addresses bug PIG-1814.
    https://issues.apache.org/jira/browse/PIG-1814


    Diffs
    -----

    http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/PigServer.java 1095577
    http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java 1095577
    http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestBZip.java 1095577

    Diff: https://reviews.apache.org/r/661/diff


    Testing
    -------

    Test-patch:
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 3 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

    Unit test:
    all pass


    Thanks,

    Daniel


    mapred.output.compress in SET statement does not work
    -----------------------------------------------------

    Key: PIG-1814
    URL: https://issues.apache.org/jira/browse/PIG-1814
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Daniel Dai
    Fix For: 0.9.0

    Attachments: PIG-1814-1.patch


    Setting output compression using "SET" in the script does not work:
    SET mapred.output.compress true;
    SET mapred.output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    We did some trick to make individual compression setting for multistore work. Instead of the above parameter, using the following works:
    SET output.compression.enabled true;
    SET output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    However, this is against intuition. We should use mapred.output.compress/mapred.output.compression.codec.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Xuefu Zhang (JIRA) at Apr 25, 2011 at 9:37 pm
    [ https://issues.apache.org/jira/browse/PIG-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13024971#comment-13024971 ]

    Xuefu Zhang commented on PIG-1814:
    ----------------------------------

    +1 patch looks good.
    mapred.output.compress in SET statement does not work
    -----------------------------------------------------

    Key: PIG-1814
    URL: https://issues.apache.org/jira/browse/PIG-1814
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Daniel Dai
    Fix For: 0.9.0

    Attachments: PIG-1814-1.patch


    Setting output compression using "SET" in the script does not work:
    SET mapred.output.compress true;
    SET mapred.output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    We did some trick to make individual compression setting for multistore work. Instead of the above parameter, using the following works:
    SET output.compression.enabled true;
    SET output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    However, this is against intuition. We should use mapred.output.compress/mapred.output.compression.codec.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Daniel Dai (JIRA) at Apr 25, 2011 at 9:43 pm
    [ https://issues.apache.org/jira/browse/PIG-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Daniel Dai resolved PIG-1814.
    -----------------------------

    Resolution: Fixed
    Hadoop Flags: [Reviewed]

    Patch committed to both trunk and 0.9 branch.
    mapred.output.compress in SET statement does not work
    -----------------------------------------------------

    Key: PIG-1814
    URL: https://issues.apache.org/jira/browse/PIG-1814
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Daniel Dai
    Fix For: 0.9.0

    Attachments: PIG-1814-1.patch


    Setting output compression using "SET" in the script does not work:
    SET mapred.output.compress true;
    SET mapred.output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    We did some trick to make individual compression setting for multistore work. Instead of the above parameter, using the following works:
    SET output.compression.enabled true;
    SET output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
    However, this is against intuition. We should use mapred.output.compress/mapred.output.compression.codec.
    --
    This message is automatically generated by JIRA.
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedJan 20, '11 at 10:24p
activeApr 25, '11 at 9:43p
posts6
users1
websitepig.apache.org

1 user in discussion

Daniel Dai (JIRA): 6 posts

People

Translate

site design / logo © 2022 Grokbase