Grokbase Groups Pig dev November 2010
FAQ
Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
-----------------------------------------------------------------------

Key: PIG-1714
URL: https://issues.apache.org/jira/browse/PIG-1714
Project: Pig
Issue Type: Bug
Reporter: Xuefu Zhang
Fix For: 0.9.0


Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.

Pig needs to clarify the right way to enable/disable compression and implement it accordingly.

The behavior change is probably related to PIg-1533.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Xuefu Zhang (JIRA) at Nov 12, 2010 at 3:53 am
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Xuefu Zhang reassigned PIG-1714:
    --------------------------------

    Assignee: Xuefu Zhang
    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.9.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Xuefu Zhang (JIRA) at Nov 12, 2010 at 3:53 am
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Xuefu Zhang updated PIG-1714:
    -----------------------------

    Attachment: jira-1714-0.patch
    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.9.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Xuefu Zhang (JIRA) at Nov 12, 2010 at 3:55 am
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Xuefu Zhang updated PIG-1714:
    -----------------------------

    Status: Patch Available (was: Open)
    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.9.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Nov 12, 2010 at 7:39 pm
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-1714:
    --------------------------------

    Fix Version/s: (was: 0.9.0)
    0.8.0
    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.8.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Xuefu Zhang (JIRA) at Nov 12, 2010 at 10:18 pm
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931543#action_12931543 ]

    Xuefu Zhang commented on PIG-1714:
    ----------------------------------

    Here is the behavior that Pig is taking:

    1. If JVM property "mapred.output.compress" is set to "true", then the output is always compressed (regardless of the output file extension).

    2. If the JVM property "mapred.output.compress" is not set or is set to "false", then whether pig output is compressed depends on the given file extension: if the extension is .bz or .bz2, then bzip compression will be used. If the extension is gz, then gzip compression will be used. In all other cases, no compression will be performed.

    3. When JVM property "mapred.output.compress" is set to "true", then another property, "mapred.output.compress.codec" must also be set. Otherwise, exception will be thrown.
    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.8.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Nov 12, 2010 at 10:28 pm
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931546#action_12931546 ]

    Richard Ding commented on PIG-1714:
    -----------------------------------

    +1. Please commit when all tests pass.
    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.8.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Xuefu Zhang (JIRA) at Nov 12, 2010 at 10:54 pm
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931558#action_12931558 ]

    Xuefu Zhang commented on PIG-1714:
    ----------------------------------

    All nightly unit test passes. Verified the fix on a real cluster and it fixes the problem as expected.
    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.8.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Xuefu Zhang (JIRA) at Nov 12, 2010 at 11:46 pm
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931570#action_12931570 ]

    Xuefu Zhang commented on PIG-1714:
    ----------------------------------

    [exec] There appear to be 463 release audit warnings before the patch and 463 release audit warnings after applying the patch.
    [exec]
    [exec]
    [exec]
    [exec]
    [exec] +1 overall.
    [exec]
    [exec] +1 @author. The patch does not contain any @author tags.
    [exec]
    [exec] +1 tests included. The patch appears to include 3 new or modified tests.
    [exec]
    [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
    [exec]
    [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
    [exec]
    [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
    [exec]
    [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
    [exec]
    [exec]
    [exec]
    [exec]
    [exec] ======================================================================
    [exec] ======================================================================
    [exec] Finished build.
    [exec] ======================================================================
    [exec] ======================================================================
    [exec]
    [exec]

    BUILD SUCCESSFUL

    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.8.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Nov 13, 2010 at 12:08 am
    [ https://issues.apache.org/jira/browse/PIG-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Richard Ding updated PIG-1714:
    ------------------------------

    Resolution: Fixed
    Hadoop Flags: [Reviewed]
    Status: Resolved (was: Patch Available)

    patch committed to both trunk and 0.8 branch. Thanks Xuefu!
    Option mapred.output.compress doesn't work in Pig 0.8 but worked in 0.7
    -----------------------------------------------------------------------

    Key: PIG-1714
    URL: https://issues.apache.org/jira/browse/PIG-1714
    Project: Pig
    Issue Type: Bug
    Reporter: Xuefu Zhang
    Assignee: Xuefu Zhang
    Fix For: 0.8.0

    Attachments: jira-1714-0.patch


    Command line options -Dmapred.output.compress and -Dmapred.output.compression.codec worked in Pig 0.7, which, when set, would compress the output, whether or not the output has an extension .gz, .bz, or .bz2. This behavior changed in 0.8 in that compression is on only if the output has such extensions. In other words, the command line options have no effect.
    Pig needs to clarify the right way to enable/disable compression and implement it accordingly.
    The behavior change is probably related to PIg-1533.
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedNov 10, '10 at 6:43p
activeNov 13, '10 at 12:08a
posts10
users1
websitepig.apache.org

1 user in discussion

Richard Ding (JIRA): 10 posts

People

Translate

site design / logo © 2022 Grokbase