Grokbase Groups Pig dev February 2011
FAQ
Job conf parameters not passed to Streaming
-------------------------------------------

Key: PIG-1869
URL: https://issues.apache.org/jira/browse/PIG-1869
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Ankur


Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.

Here is a small test script along with the streaming binary

=== stream.pig ===

A = LOAD 'dat' as (f1:chararray);
DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
B = STREAM A THROUGH MY_BIN;
dump B;

=== stream.sh ===

#!/bin/bash
export

--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

Search Discussions

  • Ankur (JIRA) at Feb 25, 2011 at 4:01 am
    [ https://issues.apache.org/jira/browse/PIG-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999209#comment-12999209 ]

    Ankur commented on PIG-1869:
    ----------------------------

    In my streaming binary, I would like to get hold of the parameter "map.input.file" that give information about the split file being processed. Also, starting with release 0.8, pig automatically combines multiple files into a single split, it is then responsibility of Pig to set this parameter correctly while reading individual split in combined split.
    Job conf parameters not passed to Streaming
    -------------------------------------------

    Key: PIG-1869
    URL: https://issues.apache.org/jira/browse/PIG-1869
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Ankur

    Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
    Here is a small test script along with the streaming binary
    === stream.pig ===
    A = LOAD 'dat' as (f1:chararray);
    DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
    B = STREAM A THROUGH MY_BIN;
    dump B;
    === stream.sh ===
    #!/bin/bash
    export
    --
    This message is automatically generated by JIRA.
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Olga Natkovich (JIRA) at Mar 2, 2011 at 1:42 am
    [ https://issues.apache.org/jira/browse/PIG-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-1869:
    --------------------------------

    Fix Version/s: 0.10
    Job conf parameters not passed to Streaming
    -------------------------------------------

    Key: PIG-1869
    URL: https://issues.apache.org/jira/browse/PIG-1869
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Ankur
    Fix For: 0.10


    Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
    Here is a small test script along with the streaming binary
    === stream.pig ===
    A = LOAD 'dat' as (f1:chararray);
    DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
    B = STREAM A THROUGH MY_BIN;
    dump B;
    === stream.sh ===
    #!/bin/bash
    export
    --
    This message is automatically generated by JIRA.
    -
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Olga Natkovich (Updated) (JIRA) at Oct 4, 2011 at 8:02 pm
    [ https://issues.apache.org/jira/browse/PIG-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Olga Natkovich updated PIG-1869:
    --------------------------------

    Fix Version/s: (was: 0.10)
    Job conf parameters not passed to Streaming
    -------------------------------------------

    Key: PIG-1869
    URL: https://issues.apache.org/jira/browse/PIG-1869
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Ankur

    Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
    Here is a small test script along with the streaming binary
    === stream.pig ===
    A = LOAD 'dat' as (f1:chararray);
    DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
    B = STREAM A THROUGH MY_BIN;
    dump B;
    === stream.sh ===
    #!/bin/bash
    export
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
    For more information on JIRA, see: http://www.atlassian.com/software/jira
  • Dmitriy V. Ryaboy (JIRA) at Sep 28, 2012 at 3:35 am
    [ https://issues.apache.org/jira/browse/PIG-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Dmitriy V. Ryaboy resolved PIG-1869.
    ------------------------------------

    Resolution: Duplicate

    Fixed in PIG-2900

    Not following what exactly you want to happen with input files, please file a separate ticket.
    Job conf parameters not passed to Streaming
    -------------------------------------------

    Key: PIG-1869
    URL: https://issues.apache.org/jira/browse/PIG-1869
    Project: Pig
    Issue Type: Bug
    Affects Versions: 0.8.0
    Reporter: Ankur

    Pig Streaming does not set the job conf parameters as environment variables for the streaming binary. This is unlike Hadoop streaming where all the job conf parameters are available as environment variables.
    Here is a small test script along with the streaming binary
    === stream.pig ===
    A = LOAD 'dat' as (f1:chararray);
    DEFINE MY_BIN `./stream.sh` SHIP('./stream.sh');
    B = STREAM A THROUGH MY_BIN;
    dump B;
    === stream.sh ===
    #!/bin/bash
    export
    --
    This message is automatically generated by JIRA.
    If you think it was sent incorrectly, please contact your JIRA administrators
    For more information on JIRA, see: http://www.atlassian.com/software/jira

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedFeb 25, '11 at 3:59a
activeSep 28, '12 at 3:35a
posts5
users1
websitepig.apache.org

1 user in discussion

Dmitriy V. Ryaboy (JIRA): 5 posts

People

Translate

site design / logo © 2022 Grokbase