Summary: Shell command execution hangs job
Issue Type: Bug
Affects Versions: 0.8.1
Executing a shell command inside a Pig script has the potential to deadlock the job. For example, the following statement will block when somebigfile.txt is sufficiently large:
%declare input `cat /path/to/somebigfile.txt`
This happens because PreprocessorContext.executeShellCommand(String) incorrectly uses Runtime.exec(). The sub-process's stderr and stdout streams should be read in a separate thread to prevent p.waitFor() from hanging when the sub-process's output is larger than the output buffer.
Per the Java Process class javadoc: "Because some native platforms only provide limited buffer size for standard input and output streams, failure to promptly write the input stream or read the output stream of the subprocess may cause the subprocess to block, and even deadlock".
See http://www.javaworld.com/jw-12-2000/jw-1229-traps.html for a correct solution.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira