FAQ
compressing intermediate results
--------------------------------

Key: PIG-467
URL: https://issues.apache.org/jira/browse/PIG-467
Project: Pig
Issue Type: Improvement
Affects Versions: types_branch
Reporter: Olga Natkovich
Fix For: types_branch


It is recommended with Hadoop 18 and later versions to compress data passed between Map and Reduce. We need to test to make sure that it gives performance gain

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Arun C Murthy (JIRA) at Sep 30, 2008 at 12:17 am
    [ https://issues.apache.org/jira/browse/PIG-467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12635619#action_12635619 ]

    Arun C Murthy commented on PIG-467:
    -----------------------------------

    It definitely is better to use lzo rather than zlib for compressing intermediate map-outputs. However, lzo might not be available on the cluster, and the right 32/64 bit lzo libraries need to installed. Hence, it would be pertinent to default to zlib but have an easy way for the user or admin to change it to lzo.
    compressing intermediate results
    --------------------------------

    Key: PIG-467
    URL: https://issues.apache.org/jira/browse/PIG-467
    Project: Pig
    Issue Type: Improvement
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Fix For: types_branch


    It is recommended with Hadoop 18 and later versions to compress data passed between Map and Reduce. We need to test to make sure that it gives performance gain
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Olga Natkovich (JIRA) at Sep 30, 2008 at 12:27 am
    [ https://issues.apache.org/jira/browse/PIG-467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-467:
    -------------------------------

    Summary: PERFORMANCE: compressing intermediate results (was: compressing intermediate results)
    PERFORMANCE: compressing intermediate results
    ---------------------------------------------

    Key: PIG-467
    URL: https://issues.apache.org/jira/browse/PIG-467
    Project: Pig
    Issue Type: Improvement
    Affects Versions: types_branch
    Reporter: Olga Natkovich
    Fix For: types_branch


    It is recommended with Hadoop 18 and later versions to compress data passed between Map and Reduce. We need to test to make sure that it gives performance gain
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedSep 30, '08 at 12:11a
activeSep 30, '08 at 12:27a
posts3
users1
websitepig.apache.org

1 user in discussion

Olga Natkovich (JIRA): 3 posts

People

Translate

site design / logo © 2022 Grokbase