[Pig-user] Passing Hadoop Site Configurations in classpath is not recommended for Local Mode

Hi,

AFAIK, the characteristic of the parameter mapred.system.dir is not conducive to be modified per job.It must be modified on mapred-default.xml, conf/mapred-site.xml.
So why local, I don't recall passing it on non-local mode either, and if passed I don't think it should work, as its final.
The parameters passed on command line usually pertain to the job performance/memory needs of the specific job.Makes sense?

Thanks & Regards,
/Rekha.

On 11/4/10 8:41 PM, "Michael Sundell" wrote:


org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Passing
Hadoop Site Configurations in classpath is not recommended for Local Mode
2010-11-04 07:29:18,971 [main] ERROR org.apache.pig.Main - ERROR 0:
mapred.system.dir:
/Users/some-name/Software/cloudera/datadir/hadoop-some-name/mapred/system
mentioned in the configuration does not exist

I found the following patch that talks about this issue and about

Looks like one of the environment variables in the pig shell script (`which pig`) is pointing to the hadoop cluster configuration. Check if environment variables PIG_CLASSPATH or PIG_CONF_DIR points to the hadoop configuration dir/file .
-Thejas



On 11/4/10 8:11 AM, "Michael Sundell" wrote:

Hi,

While trying to run Pig in local mode using "pig -x local" I get the
following error:
10/11/04 07:29:18 INFO pig.Main: Logging error messages to:
/Users/some-name/Software/cloudera/hadoop-0.20.2+737/pig_1288880958619.log
2010-11-04 07:29:18,807 [main] WARN
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Passing
Hadoop Site Configurations in classpath is not recommended for Local Mode
2010-11-04 07:29:18,971 [main] ERROR org.apache.pig.Main - ERROR 0:
mapred.system.dir:
/Users/some-name/Software/cloudera/datadir/hadoop-some-name/mapred/system
mentioned in the configuration does not exist

I found the following patch that talks about this issue and about a patch
that generates this error message:
https://issues.apache.org/jira/browse/PIG-1154

What I'm not clear about is how to address this issue.

I'm using:
pig-0.7.0+16
hadoop-0.20.2+737

Thank you

It turns out that Pig calls $HADOOP_HOME/bin/hadoop-config.sh
Inside this script this is set by default (among other things) which
causes the warning:

HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$HADOOP_HOME/conf}"

Since this is a default setting for a Hadoop installation I'm assuming
that this warning can be ignored.

/Michael

This is a defect in Pig 0.7. Pig 0.8 will automatically exclude hadoop
config file in local mode (https://issues.apache.org/jira/browse/PIG-1338)

Daniel

Michael Sundell wrote: