Grokbase › Groups › Pig › dev › November 2010
FAQ
Hadoop default config override pig.properties
---------------------------------------------

Key: PIG-1727
URL: https://issues.apache.org/jira/browse/PIG-1727
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.8.0
Reporter: Daniel Dai
Assignee: Richard Ding
Fix For: 0.8.0


In GenericOptionsParser work, we have these lines:

{code}
GenericOptionsParser parser = new GenericOptionsParser(args);
Configuration conf = parser.getConfiguration();

Properties properties = new Properties();
properties.putAll(ConfigurationUtil.toProperties(conf));
PropertiesUtil.loadDefaultProperties(properties);
{code}

conf contains two sources: one from hadoop default config files (core-site.xml, hdfs-site.xml, mapred-site.xml), the other is from hadoop related command line options. Override conf over pig property files is wrong for the default hadoop configuration part. We shall not bring default hadoop configuration in. The following code illustrate how to do that:

{code}
Configuration conf = new Configuration(false);
GenericOptionsParser parser = new GenericOptionsParser(conf, args);
conf = parser.getConfiguration();
{code}

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Search Discussions

  • Richard Ding (JIRA) at Nov 16, 2010 at 1:09 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932298#action_12932298 ]

    Richard Ding commented on PIG-1727:
    -----------------------------------

    The workaround is to use the command line option -P to pass in additional property files.
    Hadoop default config override pig.properties
    ---------------------------------------------

    Key: PIG-1727
    URL: https://issues.apache.org/jira/browse/PIG-1727
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Richard Ding
    Fix For: 0.8.0


    In GenericOptionsParser work, we have these lines:
    {code}
    GenericOptionsParser parser = new GenericOptionsParser(args);
    Configuration conf = parser.getConfiguration();
    Properties properties = new Properties();
    properties.putAll(ConfigurationUtil.toProperties(conf));
    PropertiesUtil.loadDefaultProperties(properties);
    {code}
    conf contains two sources: one from hadoop default config files (core-site.xml, hdfs-site.xml, mapred-site.xml), the other is from hadoop related command line options. Override conf over pig property files is wrong for the default hadoop configuration part. We shall not bring default hadoop configuration in. The following code illustrate how to do that:
    {code}
    Configuration conf = new Configuration(false);
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    conf = parser.getConfiguration();
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Nov 16, 2010 at 1:17 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Richard Ding updated PIG-1727:
    ------------------------------

    Attachment: PIG-1727.patch

    Attaching the patch.
    Hadoop default config override pig.properties
    ---------------------------------------------

    Key: PIG-1727
    URL: https://issues.apache.org/jira/browse/PIG-1727
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Richard Ding
    Fix For: 0.8.0

    Attachments: PIG-1727.patch


    In GenericOptionsParser work, we have these lines:
    {code}
    GenericOptionsParser parser = new GenericOptionsParser(args);
    Configuration conf = parser.getConfiguration();
    Properties properties = new Properties();
    properties.putAll(ConfigurationUtil.toProperties(conf));
    PropertiesUtil.loadDefaultProperties(properties);
    {code}
    conf contains two sources: one from hadoop default config files (core-site.xml, hdfs-site.xml, mapred-site.xml), the other is from hadoop related command line options. Override conf over pig property files is wrong for the default hadoop configuration part. We shall not bring default hadoop configuration in. The following code illustrate how to do that:
    {code}
    Configuration conf = new Configuration(false);
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    conf = parser.getConfiguration();
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • V.V.Chaitanya Krishna (JIRA) at Nov 16, 2010 at 3:29 am ⇧
    [ https://issues.apache.org/jira/browse/PIG-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932322#action_12932322 ]

    V.V.Chaitanya Krishna commented on PIG-1727:
    --------------------------------------------

    Richard,

    I see that this has more to do with the order in which properties are loaded. If we make sure that hadoop-specific default properties are loaded first, followed by the pig-specific ones, this problem should not arise. Looking at the following code snippet in Main.java :

    {noformat}
    PropertiesUtil.loadDefaultProperties(properties);
    properties.putAll(ConfigurationUtil.toProperties(conf));
    {noformat}

    The order of loading the properties object in this case seems to be such that pig-specific default properties (i.e., properties in pig-default.properties, .pigrc and pig.properties) get loaded first, followed by the ones in hadoop-default configuration. Due to this order, some of the pig properties might get overriden. Therefore, a simple swap of the above two lines to change the order of loading of properties should solve the problem.

    Thoughts?
    Hadoop default config override pig.properties
    ---------------------------------------------

    Key: PIG-1727
    URL: https://issues.apache.org/jira/browse/PIG-1727
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Richard Ding
    Fix For: 0.8.0

    Attachments: PIG-1727.patch


    In GenericOptionsParser work, we have these lines:
    {code}
    GenericOptionsParser parser = new GenericOptionsParser(args);
    Configuration conf = parser.getConfiguration();
    Properties properties = new Properties();
    properties.putAll(ConfigurationUtil.toProperties(conf));
    PropertiesUtil.loadDefaultProperties(properties);
    {code}
    conf contains two sources: one from hadoop default config files (core-site.xml, hdfs-site.xml, mapred-site.xml), the other is from hadoop related command line options. Override conf over pig property files is wrong for the default hadoop configuration part. We shall not bring default hadoop configuration in. The following code illustrate how to do that:
    {code}
    Configuration conf = new Configuration(false);
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    conf = parser.getConfiguration();
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Nov 16, 2010 at 5:35 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932550#action_12932550 ]

    Richard Ding commented on PIG-1727:
    -----------------------------------

    Thanks for the comment. The purpose of using GenericOptionsParser is to parse command line arguments generic to the Hadoop framework so that users can override system properties (including pig properties using -D option) from the command line. This is the reason for the current load ordering. As Daniel pointed out, Pig doesn't need to load the hadoop-default properties at this point. The patch now only add the command line arguments to the properties.
    Hadoop default config override pig.properties
    ---------------------------------------------

    Key: PIG-1727
    URL: https://issues.apache.org/jira/browse/PIG-1727
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Richard Ding
    Fix For: 0.8.0

    Attachments: PIG-1727.patch


    In GenericOptionsParser work, we have these lines:
    {code}
    GenericOptionsParser parser = new GenericOptionsParser(args);
    Configuration conf = parser.getConfiguration();
    Properties properties = new Properties();
    properties.putAll(ConfigurationUtil.toProperties(conf));
    PropertiesUtil.loadDefaultProperties(properties);
    {code}
    conf contains two sources: one from hadoop default config files (core-site.xml, hdfs-site.xml, mapred-site.xml), the other is from hadoop related command line options. Override conf over pig property files is wrong for the default hadoop configuration part. We shall not bring default hadoop configuration in. The following code illustrate how to do that:
    {code}
    Configuration conf = new Configuration(false);
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    conf = parser.getConfiguration();
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Daniel Dai (JIRA) at Nov 17, 2010 at 6:33 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933069#action_12933069 ]

    Daniel Dai commented on PIG-1727:
    ---------------------------------

    +1 for the patch.
    Hadoop default config override pig.properties
    ---------------------------------------------

    Key: PIG-1727
    URL: https://issues.apache.org/jira/browse/PIG-1727
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Richard Ding
    Fix For: 0.8.0

    Attachments: PIG-1727.patch


    In GenericOptionsParser work, we have these lines:
    {code}
    GenericOptionsParser parser = new GenericOptionsParser(args);
    Configuration conf = parser.getConfiguration();
    Properties properties = new Properties();
    properties.putAll(ConfigurationUtil.toProperties(conf));
    PropertiesUtil.loadDefaultProperties(properties);
    {code}
    conf contains two sources: one from hadoop default config files (core-site.xml, hdfs-site.xml, mapred-site.xml), the other is from hadoop related command line options. Override conf over pig property files is wrong for the default hadoop configuration part. We shall not bring default hadoop configuration in. The following code illustrate how to do that:
    {code}
    Configuration conf = new Configuration(false);
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    conf = parser.getConfiguration();
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Daniel Dai (JIRA) at Nov 17, 2010 at 6:37 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933071#action_12933071 ]

    Daniel Dai commented on PIG-1727:
    ---------------------------------

    To V.V.Chaitanya,
    We will merge with hadoop default configurations anyway in HExecutionEngine.init().
    Hadoop default config override pig.properties
    ---------------------------------------------

    Key: PIG-1727
    URL: https://issues.apache.org/jira/browse/PIG-1727
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Richard Ding
    Fix For: 0.8.0

    Attachments: PIG-1727.patch


    In GenericOptionsParser work, we have these lines:
    {code}
    GenericOptionsParser parser = new GenericOptionsParser(args);
    Configuration conf = parser.getConfiguration();
    Properties properties = new Properties();
    properties.putAll(ConfigurationUtil.toProperties(conf));
    PropertiesUtil.loadDefaultProperties(properties);
    {code}
    conf contains two sources: one from hadoop default config files (core-site.xml, hdfs-site.xml, mapred-site.xml), the other is from hadoop related command line options. Override conf over pig property files is wrong for the default hadoop configuration part. We shall not bring default hadoop configuration in. The following code illustrate how to do that:
    {code}
    Configuration conf = new Configuration(false);
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    conf = parser.getConfiguration();
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.
  • Richard Ding (JIRA) at Nov 17, 2010 at 6:51 pm ⇧
    [ https://issues.apache.org/jira/browse/PIG-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

    Richard Ding resolved PIG-1727.
    -------------------------------

    Resolution: Fixed
    Hadoop Flags: [Reviewed]

    patch committed to both trunk and 0.8 branch.
    Hadoop default config override pig.properties
    ---------------------------------------------

    Key: PIG-1727
    URL: https://issues.apache.org/jira/browse/PIG-1727
    Project: Pig
    Issue Type: Bug
    Components: impl
    Affects Versions: 0.8.0
    Reporter: Daniel Dai
    Assignee: Richard Ding
    Fix For: 0.8.0

    Attachments: PIG-1727.patch


    In GenericOptionsParser work, we have these lines:
    {code}
    GenericOptionsParser parser = new GenericOptionsParser(args);
    Configuration conf = parser.getConfiguration();
    Properties properties = new Properties();
    properties.putAll(ConfigurationUtil.toProperties(conf));
    PropertiesUtil.loadDefaultProperties(properties);
    {code}
    conf contains two sources: one from hadoop default config files (core-site.xml, hdfs-site.xml, mapred-site.xml), the other is from hadoop related command line options. Override conf over pig property files is wrong for the default hadoop configuration part. We shall not bring default hadoop configuration in. The following code illustrate how to do that:
    {code}
    Configuration conf = new Configuration(false);
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    conf = parser.getConfiguration();
    {code}
    --
    This message is automatically generated by JIRA.
    -
    You can reply to this email to add a comment to the issue online.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupdev @
categoriespig, hadoop
postedNov 15, '10 at 11:08p
activeNov 17, '10 at 6:51p
posts8
users1
websitepig.apache.org

1 user in discussion

Richard Ding (JIRA): 8 posts

Content

People

Support

Translate

site design / logo © 2023 Grokbase