|| at Dec 3, 2009 at 1:25 pm
sJobConf is null during the planning stage; it is defined in the
execution stage. If you are writing a LoadFunc, you can piggyback on
the DataStorage object that is passed in to determineSchema() to work
with the FS at the planning stage. I am not sure at the moment how to
work with the dfs in an EvalFunc in the planning stage. In the
execution stage, you can use sJobConf:
PigContext pigContext = new PigContext(ExecType.MAPREDUCE,
DataStorage dfs = pigContext.getDfs();
For how to use DataStorage, check out what I am doing in PIG-760 in
On Thu, Dec 3, 2009 at 7:38 AM, Tamir Kamara wrote:
As I wrote before, I saw in the source code that pig uses that syntax for
accessing the dfs.
Can you suggest a way to check if a path is a file or directory, and if a
directory to enumerate the files in it ?
On Thu, Dec 3, 2009 at 1:22 PM, Jeff Zhang wrote:
PigMapReduce.sJobConf is Null. The PigMapReduce should only been used by
hadoop internal. Do not sure why you want to use it like that.
You can look into the source code, the sJobConf will been assigned value
each time when a mapper or reducer task is initialized.
On Thu, Dec 3, 2009 at 2:50 PM, Tamir Kamara <firstname.lastname@example.org>
I'm trying to use FileLocalizer in a UDF to check if a path passed in as a
parameter is a file or a directory.
I saw in some of the pig interval code that something like this:
PigContext pc = (PigContext)
if (FileLocalizer.isFile(Path, pc)) ...
But I'm getting a NullPointerExecption. Probably I missed something.
Could someone provide an example of how to do this ?
Also, Is it possible to get a list of files in a dfs directory somehow?